{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "题目：HW2:Logistic 回归& SVM\n",
    "\n",
    "任务描述：\n",
    "\n",
    "请在Pima Indians Diabetes Data Set(皮马印第安人糖尿病数据采集）进行分类器练习\n",
    "\n",
    "\n",
    "需要提交代码文件，并给出必要的结果解释\n",
    "\n",
    "1)训练数据和测试数据分割（随机选择20%的数据作为测试集）\n",
    "\n",
    "2)适当的特征工程（及数据探索）\n",
    "\n",
    "3)Logistic 回归，并选择最佳的正则参数（L1/L2)及正则参数\n",
    "\n",
    "4)线性SVM，并选择最佳正则参数，比较与Logistic回归的性能，简单说明原因\n",
    "\n",
    "5)RBF核的SVM，并选择最佳的超参数（正则参数、RBF核函数宽度）\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 首先 import 必要的模块\n",
    "import pandas as pd \n",
    "import numpy as np \n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "#竞赛的评价指标为logloss\n",
    "from sklearn.metrics import log_loss  \n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "from matplotlib import pyplot \n",
    "import seaborn as sns\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "读取数据 & 数据探索"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Pregnancies</th>\n",
       "      <th>Glucose</th>\n",
       "      <th>BloodPressure</th>\n",
       "      <th>SkinThickness</th>\n",
       "      <th>Insulin</th>\n",
       "      <th>BMI</th>\n",
       "      <th>DiabetesPedigreeFunction</th>\n",
       "      <th>Age</th>\n",
       "      <th>Outcome</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6</td>\n",
       "      <td>148</td>\n",
       "      <td>72</td>\n",
       "      <td>35</td>\n",
       "      <td>0</td>\n",
       "      <td>33.6</td>\n",
       "      <td>0.627</td>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>85</td>\n",
       "      <td>66</td>\n",
       "      <td>29</td>\n",
       "      <td>0</td>\n",
       "      <td>26.6</td>\n",
       "      <td>0.351</td>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>183</td>\n",
       "      <td>64</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>23.3</td>\n",
       "      <td>0.672</td>\n",
       "      <td>32</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>89</td>\n",
       "      <td>66</td>\n",
       "      <td>23</td>\n",
       "      <td>94</td>\n",
       "      <td>28.1</td>\n",
       "      <td>0.167</td>\n",
       "      <td>21</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>137</td>\n",
       "      <td>40</td>\n",
       "      <td>35</td>\n",
       "      <td>168</td>\n",
       "      <td>43.1</td>\n",
       "      <td>2.288</td>\n",
       "      <td>33</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Pregnancies  Glucose  BloodPressure  SkinThickness  Insulin   BMI  \\\n",
       "0            6      148             72             35        0  33.6   \n",
       "1            1       85             66             29        0  26.6   \n",
       "2            8      183             64              0        0  23.3   \n",
       "3            1       89             66             23       94  28.1   \n",
       "4            0      137             40             35      168  43.1   \n",
       "\n",
       "   DiabetesPedigreeFunction  Age  Outcome  \n",
       "0                     0.627   50        1  \n",
       "1                     0.351   31        0  \n",
       "2                     0.672   32        1  \n",
       "3                     0.167   21        0  \n",
       "4                     2.288   33        1  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 读取数据\n",
    "# path to where the data lies\n",
    "#dpath = './data/'\n",
    "data = pd.read_csv(\"diabetes.csv\")\n",
    "data.head()\n",
    "#------------------------------\n",
    "#字段说明\n",
    "#Pregnancies:怀孕次数\n",
    "#Glucose : 口服葡萄糖耐受实验中，2小时的血浆葡萄糖浓度\n",
    "#BloodPressure : 舒张压（mm Hg)\n",
    "#SkinThickness : 三头肌皮肤褶层厚度（mm)\n",
    "#Insulin : 2小时血清胰岛素含量（uU/ml)\n",
    "#BMI : 体重指数（体重，kg/(身高，m)^2)\n",
    "\n",
    "#2)DiabetesPedigreeFunction: 糖尿病家族史\n",
    "#3)Age:年龄（岁）\n",
    "#Outcome: 输出变了/类别标签（0或1,出现糖尿病为1,否则为0)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DiabetesPedigreeFunction 的单位是什么单位，还是指犯病概率\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "数据基本信息"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 768 entries, 0 to 767\n",
      "Data columns (total 9 columns):\n",
      "Pregnancies                 768 non-null int64\n",
      "Glucose                     768 non-null int64\n",
      "BloodPressure               768 non-null int64\n",
      "SkinThickness               768 non-null int64\n",
      "Insulin                     768 non-null int64\n",
      "BMI                         768 non-null float64\n",
      "DiabetesPedigreeFunction    768 non-null float64\n",
      "Age                         768 non-null int64\n",
      "Outcome                     768 non-null int64\n",
      "dtypes: float64(2), int64(7)\n",
      "memory usage: 54.1 KB\n"
     ]
    }
   ],
   "source": [
    "data.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从数据基本信息可看出，数据还算可以，没有缺失值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Pregnancies</th>\n",
       "      <th>Glucose</th>\n",
       "      <th>BloodPressure</th>\n",
       "      <th>SkinThickness</th>\n",
       "      <th>Insulin</th>\n",
       "      <th>BMI</th>\n",
       "      <th>DiabetesPedigreeFunction</th>\n",
       "      <th>Age</th>\n",
       "      <th>Outcome</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>3.845052</td>\n",
       "      <td>120.894531</td>\n",
       "      <td>69.105469</td>\n",
       "      <td>20.536458</td>\n",
       "      <td>79.799479</td>\n",
       "      <td>31.992578</td>\n",
       "      <td>0.471876</td>\n",
       "      <td>33.240885</td>\n",
       "      <td>0.348958</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>3.369578</td>\n",
       "      <td>31.972618</td>\n",
       "      <td>19.355807</td>\n",
       "      <td>15.952218</td>\n",
       "      <td>115.244002</td>\n",
       "      <td>7.884160</td>\n",
       "      <td>0.331329</td>\n",
       "      <td>11.760232</td>\n",
       "      <td>0.476951</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.078000</td>\n",
       "      <td>21.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>99.000000</td>\n",
       "      <td>62.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>27.300000</td>\n",
       "      <td>0.243750</td>\n",
       "      <td>24.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>3.000000</td>\n",
       "      <td>117.000000</td>\n",
       "      <td>72.000000</td>\n",
       "      <td>23.000000</td>\n",
       "      <td>30.500000</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>0.372500</td>\n",
       "      <td>29.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>6.000000</td>\n",
       "      <td>140.250000</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>127.250000</td>\n",
       "      <td>36.600000</td>\n",
       "      <td>0.626250</td>\n",
       "      <td>41.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>17.000000</td>\n",
       "      <td>199.000000</td>\n",
       "      <td>122.000000</td>\n",
       "      <td>99.000000</td>\n",
       "      <td>846.000000</td>\n",
       "      <td>67.100000</td>\n",
       "      <td>2.420000</td>\n",
       "      <td>81.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Pregnancies     Glucose  BloodPressure  SkinThickness     Insulin  \\\n",
       "count   768.000000  768.000000     768.000000     768.000000  768.000000   \n",
       "mean      3.845052  120.894531      69.105469      20.536458   79.799479   \n",
       "std       3.369578   31.972618      19.355807      15.952218  115.244002   \n",
       "min       0.000000    0.000000       0.000000       0.000000    0.000000   \n",
       "25%       1.000000   99.000000      62.000000       0.000000    0.000000   \n",
       "50%       3.000000  117.000000      72.000000      23.000000   30.500000   \n",
       "75%       6.000000  140.250000      80.000000      32.000000  127.250000   \n",
       "max      17.000000  199.000000     122.000000      99.000000  846.000000   \n",
       "\n",
       "              BMI  DiabetesPedigreeFunction         Age     Outcome  \n",
       "count  768.000000                768.000000  768.000000  768.000000  \n",
       "mean    31.992578                  0.471876   33.240885    0.348958  \n",
       "std      7.884160                  0.331329   11.760232    0.476951  \n",
       "min      0.000000                  0.078000   21.000000    0.000000  \n",
       "25%     27.300000                  0.243750   24.000000    0.000000  \n",
       "50%     32.000000                  0.372500   29.000000    0.000000  \n",
       "75%     36.600000                  0.626250   41.000000    1.000000  \n",
       "max     67.100000                  2.420000   81.000000    1.000000  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#各属性的统计特性\n",
    "data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此处得到各属性的样本数目、均值、标准差、最小值、1/4分位数（25%）、中位数（50%）、3/4分位数（75%）、最大值\n",
    "可初步了解各特征的分布\n",
    "\n",
    "SkinThickness 与BloodPressure都出现了数据为0的状况,考虑现实因素，可能是统计问题，这些值不应该为0,后期要对其进行处理，可进行众数进行填充，但好像有些困难，不知道怎末进行数据处理，而Insulin最大值为846，对平均值影响太大，年龄也是如此\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAE+JJREFUeJzt3X+0ZWV93/H3BwbUoHH4MbIow2SwjmnJUpGMFEOrEW0q2jJIxcSyZIKzOsmqURLS1NGkksTYaCJiqC3trGAYLIUgFRmUxkxHwBUTKDMoIKJlpARmYZkB+aUULPDtH+e5cBj23LtnhnPP4d73a62zzt7Pefa53+sa7se9n72fJ1WFJEk72mvcBUiSJpMBIUnqZEBIkjoZEJKkTgaEJKmTASFJ6mRASJI6GRCSpE4GhCSp04JxF7AnDjrooFq6dOm4y5Ck55XNmzffW1WLZur3vA6IpUuXsmnTpnGXIUnPK0n+tk8/LzFJkjoZEJKkTgaEJKmTASFJ6mRASJI6jTQgktyR5OYk30yyqbUdkGRDktva+/6tPUnOSbIlyU1JjhplbZKk6c3GGcSbqurIqlre9tcAG6tqGbCx7QMcDyxrr9XAubNQmyRpJ8ZxiWkFsK5trwNOHGq/oAauBRYmOWQM9UmSGH1AFPCXSTYnWd3aDq6q7wO095e19kOBu4aO3draJEljMOonqY+tqruTvAzYkOQ70/RNR1s9q9MgaFYDLFmyZI8L/NnfumCPv0Nzz+Y/PnXcJUhjN9IziKq6u71vAy4Djgbumbp01N63te5bgcOGDl8M3N3xnWuranlVLV+0aMapRCRJu2lkAZFkvyQvmdoGfgH4FrAeWNm6rQQub9vrgVPb3UzHAA9OXYqSJM2+UV5iOhi4LMnUz/mvVfUXSa4HLkmyCrgTOLn1vxJ4G7AFeAQ4bYS1SZJmMLKAqKrbgdd0tN8HvLmjvYD3jaoeSdKu8UlqSVInA0KS1MmAkCR1MiAkSZ0MCElSJwNCktTJgJAkdTIgJEmdDAhJUicDQpLUyYCQJHUyICRJnQwISVInA0KS1MmAkCR1MiAkSZ0MCElSJwNCktTJgJAkdTIgJEmdDAhJUicDQpLUyYCQJHUyICRJnQwISVInA0KS1MmAkCR1MiAkSZ0MCElSJwNCktTJgJAkdTIgJEmdRh4QSfZO8o0kX2r7hye5LsltSf48yb6t/QVtf0v7fOmoa5Mk7dxsnEGcDtw6tP8J4OyqWgbcD6xq7auA+6vqFcDZrZ8kaUxGGhBJFgNvB/607Qc4Dri0dVkHnNi2V7R92udvbv0lSWMw6jOITwP/Bniy7R8IPFBVj7f9rcChbftQ4C6A9vmDrb8kaQxmDIgk+yXZq22/MskJSfbpcdw/BbZV1ebh5o6u1eOz4e9dnWRTkk3bt2+fqQxJ0m7qcwbxNeCFSQ4FNgKnAef3OO5Y4IQkdwAXM7i09GlgYZIFrc9i4O62vRU4DKB9/lLgBzt+aVWtrarlVbV80aJFPcqQJO2OPgGRqnoEOAn491X1DuCImQ6qqg9V1eKqWgr8EvDVqjoFuAp4Z+u2Eri8ba9v+7TPv1pVzzqDkCTNjl4BkeT1wCnAl1vbgmn6z+SDwBlJtjAYYzivtZ8HHNjazwDW7MHPkCTtoT5/6H8d+BBwWVXdkuTlDM4Cequqq4Gr2/btwNEdfR4FTt6V75Ukjc6MAVFV1wDXJNmv7d8OfGDUhUmSxqvPXUyvT/Jt2sNuSV6T5D+OvDJJ0lj1GYP4NPBPgPsAqupG4A2jLEqSNH69HpSrqrt2aHpiBLVIkiZIn0Hqu5L8HFBtYr0P8My5lSRJc1CfM4hfBd7HYCqMrcCRbV+SNIf1uYvpXgbPQEiS5pE+dzGtS7JwaH//JJ8dbVmSpHHrc4np1VX1wNROVd0PvHZ0JUmSJkGfgNgryf5TO0kOYM+m2pAkPQ/0+UN/FvDXSaYW+TkZ+NjoSpIkTYI+g9QXJNkMvInBmg0nVdW3R16ZJGms+l4q+g6D9aMXACRZUlV3jqwqSdLYzRgQSd4PnAncw+AJ6jBY6e3Voy1NkjROfc4gTgd+uqruG3UxkqTJ0ecupruAB0ddiCRpsvQ5g7gduDrJl4HHphqr6lMjq0qSNHZ9AuLO9tq3vSRJ80Cf21x/DyDJflX1o9GXJEmaBK4oJ0nq5IpykqROrignSerkinKSpE6uKCdJ6jTtGUSSvYH3VJUryknSPDPtGURVPQGsmKVaJEkTpM8YxNeTfAb4c+Cp5yCq6oaRVSVJGrs+AfFz7f33h9oKOO65L0eSNClmGoPYCzi3qi6ZpXokSRNipjGIJ4Ffm6VaJEkTpM9trhuS/OskhyU5YOo18sokSWPVZwzive19+NmHAl7+3JcjSZoUfWZzPXw2CpEkTZY+a1Kf2tVeVRfMcNwLga8BL2g/59KqOjPJ4cDFwAHADQwexPtxkhcAFwA/y2BiwF+sqjt24XeRJD2H+oxBvG7o9Y+A3wVO6HHcY8BxVfUaBtNzvDXJMcAngLOrahlwP7Cq9V8F3F9VrwDObv0kSWPS5xLT+4f3k7wU+FyP4wr4Ydvdp72mnp/4F619HYPAOZfBE9u/29ovBT6TJO17JEmzrNd03zt4BFjWp2OSvZN8E9gGbAC+BzxQVY+3LlsZTAJIe78LoH3+IHDgbtQnSXoO9BmDuILB//OHQaAcAfR6cK7N5XRkkoXAZcDf7+o29aOm+Wy4ntXAaoAlS5b0KUOStBv63Ob6yaHtx4G/raqtu/JDquqBJFcDxwALkyxoZwmLgbtbt63AYcDWJAuAlwI/6PiutcBagOXLl3v5SZJGpM8lpjuB66rqmqr6OnBfkqUzHZRkUTtzIMmLgLcwWGjoKuCdrdtK4PK2vb7t0z7/quMPkjQ+fQLi88CTQ/tPtLaZHAJcleQm4HpgQ1V9CfggcEaSLQzGGM5r/c8DDmztZwBr+v0KkqRR6HOJaUFV/Xhqpz2zsO9MB1XVTcBrO9pvB47uaH8UOLlHPZKkWdDnDGJ7kqeee0iyArh3dCVJkiZBnzOIXwUubIsGwWAwufPpaknS3NHnQbnvAcckeTGQqnp49GVJksZtxktMSf5dkoVV9cOqejjJ/kn+YDaKkySNT58xiOOr6oGpnaq6H3jb6EqSJE2CPgGxd5tpFXjqmYYXTNNfkjQH9Bmk/i/AxiR/xmDqi/cymGRPkjSH9Rmk/qP2sNtbWtNHq+oroy1LkjRufc4gAL7B09N1f2N05Uiacufvv2rcJWgCLfnIzbP2s/rcxfQu4H8ymB/pXcB1Sd45/VGSpOe7PmcQvw28rqq2wWASPuB/MFjUR5I0R/W5i2mvqXBo7ut5nCTpeazPGcRfJPkKcFHb/0XgytGVJEmaBH3uYvqtJCcB/5DBqm9rq+qykVcmSRqrXncxVdUXgC+MuBZJ0gRxLEGS1MmAkCR12mlAJNnY3j8xe+VIkibFdGMQhyR5I3BCkosZDFA/papuGGllkqSxmi4gPgKsARYDn9rhswKOG1VRkqTx22lAVNWlwKVJ/m1VfXQWa5IkTYA+z0F8NMkJwBta09VV9aXRliVJGrc+k/X9IXA68O32Or21SZLmsD4Pyr0dOLKqngRIso7BlN8fGmVhkqTx6vscxMKh7ZeOohBJ0mTpcwbxh8A3klzF4FbXN+DZgyTNeX0GqS9KcjXwOgYB8cGq+j+jLkySNF59J+v7PrB+xLVIkiaIczFJkjoZEJKkTtMGRJK9knxrtoqRJE2OaQOiPftwY5Ils1SPJGlC9LnEdAhwS5KNSdZPvWY6KMlhSa5KcmuSW5Kc3toPSLIhyW3tff/WniTnJNmS5KYkR+3ZryZJ2hN97mL6vd387seB36yqG5K8BNicZAPwy8DGqvp4kjUMZoz9IHA8sKy9/gFwbnuXJI3BjGcQVXUNcAewT9u+HphxLYiq+v7UmhFV9TBwK3AosAJY17qtA05s2yuAC2rgWmBhkkN27deRJD1X+kzW9y+BS4H/3JoOBb64Kz8kyVLgtcB1wMHtuYqp5yteNvS9dw0dtrW1SZLGoM8YxPuAY4GHAKrqNp7+oz6jJC8G/hvw61X10HRdO9qq4/tWJ9mUZNP27dv7liFJ2kV9AuKxqvrx1E6SBXT84e6SZB8G4XBhVX2hNd8zdemovW9r7VuBw4YOXwzcveN3VtXaqlpeVcsXLVrUpwxJ0m7oExDXJPkw8KIk/xj4PHDFTAclCXAecGtVDS9Zuh5Y2bZXApcPtZ/a7mY6Bnhw6lKUJGn29bmLaQ2wCrgZ+BXgSuBPexx3LPAe4OYk32xtHwY+DlySZBVwJ3By++xK4G3AFuAR4LSev4MkaQT6zOb6ZFsk6DoGl5a+W1UzXmKqqr+ie1wB4M0d/YvBeIckaQLMGBBJ3g78J+B7DP7gH57kV6rqv4+6OEnS+PS5xHQW8Kaq2gKQ5O8CXwYMCEmaw/oMUm+bCofmdp6+80iSNEft9AwiyUlt85YkVwKXMBiDOJnB09SSpDlsuktM/2xo+x7gjW17O7D/yCqSJE2EnQZEVXmbqSTNY33uYjoceD+wdLh/VZ0wurIkSePW5y6mLzJ4IvoK4MnRliNJmhR9AuLRqjpn5JVIkiZKn4D4kyRnAn8JPDbVOLXWgyRpbuoTEK9iMKfScTx9ianaviRpjuoTEO8AXj485bckae7r8yT1jcDCURciSZosfc4gDga+k+R6njkG4W2ukjSH9QmIM0dehSRp4vRZD+Ka2ShEkjRZ+jxJ/TBPr0G9L7AP8KOq+slRFiZJGq8+ZxAvGd5PciJw9MgqkiRNhD53MT1DVX0Rn4GQpDmvzyWmk4Z29wKW8/QlJ0nSHNXnLqbhdSEeB+4AVoykGknSxOgzBuG6EJI0D0235OhHpjmuquqjI6hHkjQhpjuD+FFH237AKuBAwICQpDlsuiVHz5raTvIS4HTgNOBi4KydHSdJmhumHYNIcgBwBnAKsA44qqrun43CJEnjNd0YxB8DJwFrgVdV1Q9nrSpJ0thN96DcbwJ/B/gd4O4kD7XXw0kemp3yJEnjMt0YxC4/ZS1JmjsMAUlSJwNCktTJgJAkdTIgJEmdRhYQST6bZFuSbw21HZBkQ5Lb2vv+rT1JzkmyJclNSY4aVV2SpH5GeQZxPvDWHdrWABurahmwse0DHA8sa6/VwLkjrEuS1MPIAqKqvgb8YIfmFQyeyKa9nzjUfkENXAssTHLIqGqTJM1stscgDq6q7wO095e19kOBu4b6bW1tz5JkdZJNSTZt3759pMVK0nw2KYPU6WjrXLWuqtZW1fKqWr5o0aIRlyVJ89dsB8Q9U5eO2vu21r4VOGyo32Lg7lmuTZI0ZLYDYj2wsm2vBC4faj+13c10DPDg1KUoSdJ49FmTerckuQj4eeCgJFuBM4GPA5ckWQXcCZzcul8JvA3YAjzCYN0JSdIYjSwgqurdO/nozR19C3jfqGqRJO26SRmkliRNGANCktTJgJAkdTIgJEmdDAhJUicDQpLUyYCQJHUyICRJnQwISVInA0KS1MmAkCR1MiAkSZ0MCElSJwNCktTJgJAkdTIgJEmdDAhJUicDQpLUyYCQJHUyICRJnQwISVInA0KS1MmAkCR1MiAkSZ0MCElSJwNCktTJgJAkdTIgJEmdDAhJUicDQpLUyYCQJHUyICRJnSYqIJK8Ncl3k2xJsmbc9UjSfDYxAZFkb+A/AMcDRwDvTnLEeKuSpPlrYgICOBrYUlW3V9WPgYuBFWOuSZLmrUkKiEOBu4b2t7Y2SdIYLBh3AUPS0VbP6pSsBla33R8m+e5Iq5pfDgLuHXcRkyCfXDnuEvRM/tuccmbXn8pd9lN9Ok1SQGwFDhvaXwzcvWOnqloLrJ2touaTJJuqavm465B25L/N8ZikS0zXA8uSHJ5kX+CXgPVjrkmS5q2JOYOoqseT/BrwFWBv4LNVdcuYy5KkeWtiAgKgqq4Erhx3HfOYl+40qfy3OQapetY4sCRJEzUGIUmaIAaEnOJEEyvJZ5NsS/KtcdcyHxkQ85xTnGjCnQ+8ddxFzFcGhJziRBOrqr4G/GDcdcxXBoSc4kRSJwNCvaY4kTT/GBDqNcWJpPnHgJBTnEjqZEDMc1X1ODA1xcmtwCVOcaJJkeQi4G+An06yNcmqcdc0n/gktSSpk2cQkqROBoQkqZMBIUnqZEBIkjoZEJKkTgaE5r0ki5NcnuS2JN9L8iftmZDpjvnwbNUnjYsBoXktSYAvAF+sqmXAK4EXAx+b4VADQnOeAaH57jjg0ar6M4CqegL4DeC9Sf5Vks9MdUzypSQ/n+TjwIuSfDPJhe2zU5PclOTGJJ9rbT+VZGNr35hkSWs/P8m5Sa5KcnuSN7Z1D25Ncv7Qz/uFJH+T5IYkn0/y4ln7X0XCgJB+Btg83FBVDwF3spM126tqDfB/q+rIqjolyc8Avw0cV1WvAU5vXT8DXFBVrwYuBM4Z+pr9GYTTbwBXAGe3Wl6V5MgkBwG/A7ylqo4CNgFnPBe/sNRX538A0jwSumev3Vl7l+OAS6vqXoCqmlq/4PXASW37c8AfDR1zRVVVkpuBe6rqZoAktwBLGUyaeATw9cFVMPZlMOWENGsMCM13twD/fLghyU8ymOH2QZ55lv3CnXxH3zAZ7vNYe39yaHtqfwHwBLChqt7d43ulkfASk+a7jcBPJDkVnlqC9SwGS13eDhyZZK8khzFYfW/K/0uyz9B3vCvJge07Dmjtf81gdlyAU4C/2oW6rgWOTfKK9p0/keSVu/rLSXvCgNC8VoPZKt8BnJzkNuB/AY8yuEvp68D/Bm4GPgncMHToWuCmJBe22W8/BlyT5EbgU63PB4DTktwEvIenxyb61LUd+GXgonb8tcDf293fU9odzuYqSerkGYQkqZMBIUnqZEBIkjoZEJKkTgaEJKmTASFJ6mRASJI6GRCSpE7/H6faJrOoDn8hAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fd5bf2d4668>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#观察一下是否有糖尿病的人数分布\n",
    "sns.countplot(data.Outcome);\n",
    "pyplot.xlabel('Outcome');\n",
    "pyplot.ylabel('Number of occurrences');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "#数据准备\n",
    "y = data['Outcome'].values\n",
    "X = data.drop('Outcome', axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(614, 8)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#将数据分割训练数据与测试数据\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# 随机采样20%的数据构建测试样本，其余作为训练样本\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=33, test_size=0.2)\n",
    "X_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 数据标准化\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "# 初始化特征的标准化器\n",
    "ss_X = StandardScaler()\n",
    "\n",
    "# 分别对训练和测试数据的特征进行标准化处理\n",
    "X_train = ss_X.fit_transform(X_train)\n",
    "X_test = ss_X.transform(X_test)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "模型训练"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "default Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "lr= LogisticRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "logloss of each fold is:  [0.46129644 0.46510588 0.56426612 0.44377717 0.46617809]\n",
      "cv logloss is: 0.48012474004701106\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jck/.local/lib/python3.6/site-packages/sklearn/cross_validation.py:41: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.\n",
      "  \"This module will be removed in 0.20.\", DeprecationWarning)\n"
     ]
    }
   ],
   "source": [
    "# 交叉验证用于评估模型性能和进行参数调优（模型选择）\n",
    "#分类任务中交叉验证缺省是采用StratifiedKFold\n",
    "from sklearn.cross_validation import cross_val_score\n",
    "loss = cross_val_score(lr, X_train, y_train, cv=5, scoring='neg_log_loss')#5折交叉验证\n",
    "print ('logloss of each fold is: ',-loss)\n",
    "print('cv logloss is:', -loss.mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "缺省参数的Logistic回归，采用五折交叉验证，得到平均损失为0.480，总体来说还可以，但未加正则项，应该还可以进一步优化"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "正则化的 Logistic Regression及参数调优"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "logistic回归的需要调整超参数有：C（正则系数，一般在log域（取log后的值）均匀设置候选参数）和正则函数penalty（L2/L1） 目标函数为：J = sum(logloss(f(xi), yi)) + C* penalty\n",
    "\n",
    "在sklearn框架下，不同学习器的参数调整步骤相同： 设置候选参数集合 调用GridSearchCV 调用fit\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5, error_score='raise',\n",
       "       estimator=LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False),\n",
       "       fit_params=None, iid=True, n_jobs=1,\n",
       "       param_grid={'penalty': ['l1', 'l2'], 'C': [0.0001, 0.001, 0.01, 0.1, 1, 10, 100, 1000, 10000]},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n",
       "       scoring='neg_log_loss', verbose=0)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "#需要调优的参数\n",
    "# 请尝试将L1正则和L2正则分开，并配合合适的优化求解算法（slover）\n",
    "#tuned_parameters = {'penalty':['l1','l2'],\n",
    "#                   'C': [0.001, 0.01, 0.1, 1, 10, 100, 1000]\n",
    "#                   }\n",
    "penaltys = ['l1','l2']\n",
    "Cs = [ 0.0001,0.001,0.01, 0.1, 1, 10, 100,1000,10000]\n",
    "tuned_parameters = dict(penalty = penaltys, C = Cs)\n",
    "\n",
    "lr_penalty= LogisticRegression()\n",
    "grid= GridSearchCV(lr_penalty, tuned_parameters,cv=5, scoring='neg_log_loss')\n",
    "#依次为：分类器，要调整的参数，交叉验证，评价指标\n",
    "grid.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jck/.local/lib/python3.6/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/home/jck/.local/lib/python3.6/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/home/jck/.local/lib/python3.6/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/home/jck/.local/lib/python3.6/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/home/jck/.local/lib/python3.6/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/home/jck/.local/lib/python3.6/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('split4_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/home/jck/.local/lib/python3.6/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'mean_fit_time': array([0.0006043 , 0.00075755, 0.00041943, 0.00073452, 0.00196438,\n",
       "        0.00096154, 0.00063443, 0.00304046, 0.00073323, 0.00073142,\n",
       "        0.00071802, 0.000769  , 0.00069218, 0.00073767, 0.00072722,\n",
       "        0.00076885, 0.00072637, 0.00075388]),\n",
       " 'mean_score_time': array([0.00044837, 0.00052037, 0.00040641, 0.00072551, 0.00119286,\n",
       "        0.00054016, 0.00040369, 0.00043674, 0.0003933 , 0.00039353,\n",
       "        0.00040965, 0.00042386, 0.00039368, 0.00041566, 0.00039635,\n",
       "        0.0004169 , 0.00039806, 0.00041027]),\n",
       " 'mean_test_score': array([-0.69314718, -0.68659497, -0.69314718, -0.64214833, -0.6721329 ,\n",
       "        -0.52844007, -0.48659143, -0.47999943, -0.48043546, -0.48017599,\n",
       "        -0.48090065, -0.48086932, -0.48095238, -0.48095123, -0.48095536,\n",
       "        -0.48095956, -0.48095774, -0.48096039]),\n",
       " 'mean_train_score': array([-0.69314718, -0.6864897 , -0.69314718, -0.6412946 , -0.67079105,\n",
       "        -0.52380684, -0.47502544, -0.46674403, -0.46228794, -0.46214818,\n",
       "        -0.46206768, -0.46206619, -0.46206531, -0.4620653 , -0.46206529,\n",
       "        -0.46206529, -0.46206529, -0.46206529]),\n",
       " 'param_C': masked_array(data=[0.0001, 0.0001, 0.001, 0.001, 0.01, 0.01, 0.1, 0.1, 1,\n",
       "                    1, 10, 10, 100, 100, 1000, 1000, 10000, 10000],\n",
       "              mask=[False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False,\n",
       "                    False, False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'param_penalty': masked_array(data=['l1', 'l2', 'l1', 'l2', 'l1', 'l2', 'l1', 'l2', 'l1',\n",
       "                    'l2', 'l1', 'l2', 'l1', 'l2', 'l1', 'l2', 'l1', 'l2'],\n",
       "              mask=[False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False,\n",
       "                    False, False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'params': [{'C': 0.0001, 'penalty': 'l1'},\n",
       "  {'C': 0.0001, 'penalty': 'l2'},\n",
       "  {'C': 0.001, 'penalty': 'l1'},\n",
       "  {'C': 0.001, 'penalty': 'l2'},\n",
       "  {'C': 0.01, 'penalty': 'l1'},\n",
       "  {'C': 0.01, 'penalty': 'l2'},\n",
       "  {'C': 0.1, 'penalty': 'l1'},\n",
       "  {'C': 0.1, 'penalty': 'l2'},\n",
       "  {'C': 1, 'penalty': 'l1'},\n",
       "  {'C': 1, 'penalty': 'l2'},\n",
       "  {'C': 10, 'penalty': 'l1'},\n",
       "  {'C': 10, 'penalty': 'l2'},\n",
       "  {'C': 100, 'penalty': 'l1'},\n",
       "  {'C': 100, 'penalty': 'l2'},\n",
       "  {'C': 1000, 'penalty': 'l1'},\n",
       "  {'C': 1000, 'penalty': 'l2'},\n",
       "  {'C': 10000, 'penalty': 'l1'},\n",
       "  {'C': 10000, 'penalty': 'l2'}],\n",
       " 'rank_test_score': array([17, 16, 17, 14, 15, 13, 12,  1,  3,  2,  5,  4,  7,  6,  8, 10,  9,\n",
       "        11], dtype=int32),\n",
       " 'split0_test_score': array([-0.69314718, -0.68682793, -0.69314718, -0.64371675, -0.66946739,\n",
       "        -0.52816851, -0.48618352, -0.4678555 , -0.46345842, -0.46129644,\n",
       "        -0.46108543, -0.46088502, -0.46086874, -0.4608487 , -0.46084484,\n",
       "        -0.46084513, -0.46084065, -0.46084477]),\n",
       " 'split0_train_score': array([-0.69314718, -0.68637921, -0.69314718, -0.64085132, -0.66189542,\n",
       "        -0.52529099, -0.47823688, -0.47085398, -0.46684264, -0.46669378,\n",
       "        -0.46662379, -0.46662226, -0.46662151, -0.46662149, -0.46662149,\n",
       "        -0.46662148, -0.46662149, -0.46662148]),\n",
       " 'split1_test_score': array([-0.69314718, -0.68647305, -0.69314718, -0.64113725, -0.67517494,\n",
       "        -0.52518603, -0.47167174, -0.47077815, -0.46426303, -0.46510588,\n",
       "        -0.46463591, -0.46473286, -0.46468798, -0.46469927, -0.46469108,\n",
       "        -0.46469595, -0.46469636, -0.46469561]),\n",
       " 'split1_train_score': array([-0.69314718, -0.68657476, -0.69314718, -0.64188719, -0.67797602,\n",
       "        -0.52532854, -0.47970446, -0.47076994, -0.46692691, -0.46678157,\n",
       "        -0.46671611, -0.4667146 , -0.4667139 , -0.46671388, -0.46671388,\n",
       "        -0.46671388, -0.46671388, -0.46671388]),\n",
       " 'split2_test_score': array([-0.69314718, -0.68698087, -0.69314718, -0.64575466, -0.6680453 ,\n",
       "        -0.5512625 , -0.54753163, -0.54301469, -0.56469343, -0.56426612,\n",
       "        -0.56847691, -0.56834734, -0.56880003, -0.56879096, -0.56882473,\n",
       "        -0.5688357 , -0.56883367, -0.56884018]),\n",
       " 'split2_train_score': array([-0.69314718, -0.68623543, -0.69314718, -0.63908575, -0.66162164,\n",
       "        -0.5135644 , -0.4556435 , -0.44783203, -0.44194603, -0.44184498,\n",
       "        -0.44173183, -0.44173047, -0.44172922, -0.4417292 , -0.44172919,\n",
       "        -0.44172919, -0.44172919, -0.44172919]),\n",
       " 'split3_test_score': array([-0.69314718, -0.68619969, -0.69314718, -0.63856226, -0.67581323,\n",
       "        -0.50975335, -0.45503481, -0.44706595, -0.4436583 , -0.44377717,\n",
       "        -0.44397751, -0.44400066, -0.44403106, -0.4440332 , -0.44403475,\n",
       "        -0.44403656, -0.44403079, -0.4440369 ]),\n",
       " 'split3_train_score': array([-0.69314718, -0.68672568, -0.69314718, -0.64333601, -0.67874616,\n",
       "        -0.5309552 , -0.48382875, -0.47509784, -0.47066672, -0.47052445,\n",
       "        -0.4704446 , -0.47044314, -0.47044227, -0.47044226, -0.47044225,\n",
       "        -0.47044225, -0.47044225, -0.47044225]),\n",
       " 'split4_test_score': array([-0.69314718, -0.68648733, -0.69314718, -0.64152376, -0.67221592,\n",
       "        -0.52767402, -0.47216494, -0.47104103, -0.46582436, -0.46617809,\n",
       "        -0.46606782, -0.46612355, -0.46611659, -0.4661268 , -0.46612416,\n",
       "        -0.46612722, -0.46613001, -0.46612726]),\n",
       " 'split4_train_score': array([-0.69314718, -0.68653345, -0.69314718, -0.64131272, -0.67371601,\n",
       "        -0.52389506, -0.47771361, -0.46916638, -0.46505739, -0.46489609,\n",
       "        -0.46482209, -0.46482048, -0.46481966, -0.46481964, -0.46481963,\n",
       "        -0.46481963, -0.46481963, -0.46481963]),\n",
       " 'std_fit_time': array([1.40154325e-04, 1.87186227e-04, 8.96524112e-05, 1.50852045e-04,\n",
       "        1.25783935e-03, 3.34075632e-04, 4.84030820e-05, 4.64162418e-03,\n",
       "        4.59285036e-05, 3.30449871e-05, 3.04353620e-05, 3.45111532e-05,\n",
       "        3.08544962e-05, 4.15929394e-05, 4.53239676e-05, 2.78643497e-05,\n",
       "        2.73059957e-05, 6.41864262e-05]),\n",
       " 'std_score_time': array([2.43404702e-05, 1.19140090e-04, 4.73978534e-05, 6.03152224e-04,\n",
       "        6.97537035e-04, 1.52049052e-04, 9.06417144e-06, 5.03133028e-05,\n",
       "        1.07225824e-05, 1.65625253e-05, 3.08205791e-05, 3.79094238e-05,\n",
       "        1.94426806e-05, 1.77781622e-05, 1.98215190e-05, 2.54894328e-05,\n",
       "        2.04873572e-06, 2.54281658e-05]),\n",
       " 'std_test_score': array([0.        , 0.0002777 , 0.        , 0.00243715, 0.00305426,\n",
       "        0.0132657 , 0.03205927, 0.03276814, 0.0429414 , 0.04285084,\n",
       "        0.04453439, 0.04448689, 0.04466543, 0.04466183, 0.044676  ,\n",
       "        0.04467945, 0.04467979, 0.04468121]),\n",
       " 'std_train_score': array([0.        , 0.00016834, 0.        , 0.00138524, 0.00757197,\n",
       "        0.00566626, 0.00992563, 0.00965834, 0.01033376, 0.01031565,\n",
       "        0.01033124, 0.01033118, 0.01033136, 0.01033136, 0.01033136,\n",
       "        0.01033136, 0.01033136, 0.01033136])}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# view the complete results (list of named tuples)\n",
    "grid.cv_results_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.4799994349368312\n",
      "{'C': 0.1, 'penalty': 'l2'}\n"
     ]
    }
   ],
   "source": [
    "# examine the best model\n",
    "print(-grid.best_score_)\n",
    "print(grid.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果最佳值在候选参数的边缘，最好再尝试更大的候选参数或更小的候选参数，直到找到拐点。 l2, c=0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jck/.local/lib/python3.6/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/home/jck/.local/lib/python3.6/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZQAAAEKCAYAAAA1qaOTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xd8VGX2+PHPSSEFQksgQCKEUJSaIAgIiIqKWBbFgq4iiiBr+y2uZdV1dXXLd3FZy65lRVFERcGGYKUJUqQYkY4CkgAJoYUkkJ6ZeX5/zE0IkJAJzMydDOf9eg1z+z2TkJw897n3PGKMQSmllDpdIXYHoJRSKjhoQlFKKeUVmlCUUkp5hSYUpZRSXqEJRSmllFdoQlFKKeUVmlCUUkp5hSYUpZRSXqEJRSmllFeE2R2AP8XFxZmkpCS7w1BKqXrlxx9/PGiMaVHbdmdUQklKSiItLc3uMJRSql4RkZ2ebKeXvJRSSnmFJhSllFJeoQlFKaWUV5xRfSjVKS8vJzMzk5KSErtDCViRkZEkJiYSHh5udyhKqQB2xieUzMxMYmJiSEpKQkTsDifgGGPIyckhMzOT9u3b2x2OUiqAnfGXvEpKSoiNjdVkUgMRITY2VltwSqlanfEJBdBkUgv9+iilPKEJ5RTcNHkFN01eYXcYSikVUDShBIBGjRpVTg8bNoymTZty9dVXV7vtfffdR2pqKl27diUqKorU1FRSU1P5+OOP63TONWvW8M0335xW3Cq49Zt6Pf2mXm93GCfQuOrGn3Gd8Z3ygeaRRx6hqKiIyZMnV7v+lVdeASAjI4Orr76atWvXntJ51qxZw8aNGxk2bNgpx6q8o+KHfdWYT2yORKnToy2UAHPJJZcQExNzSvtu27aNyy+/nN69ezN48GC2bt0KwIwZM+jevTspKSlcfPHFFBcX89e//pXp06efUutGKaWqoy2UKp75fBOb9xyudbvN2e5tPOlH6dqmMX/5TbfTjs0T48ePZ8qUKXTo0IHly5dz//33M2/ePJ555hkWL15MfHw8eXl5REVF8dRTT7Fx40ZefPFFv8SmlAp+mlCCRF5eHitXruT6649eK3U4HAAMHDiQ0aNHc+ONN3LdddfZFaJSKshpQqnC05ZERctk5u/O92U4dWKMIS4urto+lTfeeINVq1bxxRdfkJKSwvr1622IUCkV7LQPJUg0a9aM1q1bM2vWLABcLhfr1q0DYMeOHfTv35+//e1vNGvWjKysLGJiYjhy5IidISulgowtCUVEmovIfBHZZr03O8m2jUUkS0RerrJssYj8IiJrrVdL/0TuexdccAE33ngjCxcuJDExkblz53q874wZM3jttddISUmhW7dufPHFFwD84Q9/oEePHvTo0YNLL72U7t27M2TIENatW0evXr20U14p5RV2XfJ6DFhojJkoIo9Z84/WsO3fgO+qWX6rMSYoRssqKCionF66dKlH+yQlJbFx48ZjliUnJ1ebgObMmXPCshYtWuhgY0opr7IroVwDXGRNTwMWU01CEZHeQDzwDdDHT7HVKpD6TpTnxnwzBoCpw6baHIlSwcmuPpR4Y0w2gPV+wiUrEQkBngMeqeEYU63LXU+KFptSSinb+ayFIiILgFbVrHrCw0PcC3xljNldTb641RiTJSIxwCfAbcA7NcQxHhgP0LZtWw9PrZRSqq58llCMMZfWtE5E9olIa2NMtoi0BvZXs9n5wAUici/QCGggIgXGmMeMMVnWOY6IyPtAX2pIKMaY14HXAfr06WNO71Mp5X1/mb7FPTHG3jiOp3HVjcZl3yWvOcDt1vTtwOzjNzDG3GqMaWuMSQIeBt4xxjwmImEiEgcgIuHA1cDG4/dXSinlX3Z1yk8EPhSRscAu4EYAEekD3G2MGXeSfSOAuVYyCQUWAG/4ON5jTb3K/T7mS7+eVp2eipI5SinfsKWFYozJMcZcYozpZL0fspanVZdMjDFvG2Put6YLjTG9jTE9jTHdjDETjDFOf38Gb/J3+fpZs2YxadKk045bKaWq0tIrAcZb5esdDgdhYdV/e0eMGOGdYOuZx97PcE8E2DVupYKFll4JMKdTvn7QoEE88cQTDB48mJdffpnZs2fTr18/evXqxdChQ9m/333vw5QpU3jggQcAGDVqFBMmTGDAgAEkJydXlm5RSqm60hZKVV8/Bns31L7dXqu4YkVfysm06gFXTDy9uOrg8OHDLFmyBIDc3FyGDx+OiPDaa6/x3HPP8eyzz56wz/79+1m+fDkbNmxg5MiRZ2wLRil1ejShBJmbb765cnrXrl2MHDmSvXv3UlpaSufOnavd59prr0VE6NmzJ1lZWf4K9YxjjMGVn4/j4EHrlYNjz04SS8tpIC6yxw0HYwDjfsPACe+m4mDVv1fZtvIYJ93++GVH9+tcVgpA5g0DvfUl8AqNq24q4irfvo7wjik+PZcmlKo8bUkE8F1eDRs2rJy+7777+NOf/sSVV17JggULmDix+s8XERFROW2MPqpTV67CwqNJ4oD1nnMQZ9X5gwdx5ORAefkJ+zcICSWkgVCQ9rN7QXV1H+SEicrJ6laduLJi+uiCE58XlqOrBUy5+4p4aVZuNQHZR+Oqm4q4THGhz8+lCSWI5efnk5CQgDGGadOm2R1OveIqLXUnhJycKoniAI6DB3EezKnSyjiIKS4+8QAhIYTGNicsNpawhqFEtIsirG0EYeWHCI0sJ6xhGGHJPQjregGrXptKfmE4V8zbXN1vedt8dWlXAK5csNnmSI6lcdVNRVydegzw+bk0oQSYCy64gJ9//pmCggISExN58803ufzyy0/pWE8//TQjRowgMTGRvn37kp2d7eVo65cIVxGhLijZvNmdKCpbDwfcyaNKonAdrv6ZldCmTQlrEUdobBxRKSmExcUR1iKOsDj3srBmTQhzZBKaswbZuQwyl4OzDELCIPE8aD8C2g+GhD4QHglA3r+nuw8eQMlEqVOhCSUAeKt8/bJly46Zv/76648ZErjCuHFHH/V57733aowlmBiXi7hciCmC9OuO/ZqENGzoTggt4ojo3JmG559/NEnExREWayWN5s2RBg2OPbDLCdlrIX0JpL8N362E8iJAoE0q9L/HnUDO6g8RjVAqmGlCORUB2HeiTu7Af/9LTBHkN4SuE/9rtSxaEBYbS0hUlOcHcrngwBYrgSyBjOVQmu9e17IrnDvanUDaDYCoGseNUyooaUJRQS/v44/JeW0yhxvCoabQ+LLLPN/ZGDi0A9K/s5LIUig66F7XPBm6W5ewki6ARkEzcKhSp0QTigpqBcuWk/2Xp2k4aBDp6cs866fI2221Ppa63w9bt1I3ToBOlx1NIE3P8m3wStUzmlBU0Cr5ZStZEyYQ0bEjCS++wMZr+1a/YcH+o5ew0pdAbrp7eXScO3m0vwDaX+hukWjHuVI10oSiglL5vv3svvtuQho25KzJrxFapQAnxbnuvo+KBHLAGi8iogkkDYJ+d7sTScsufkkgz9zaBYArfX6mutG46kbj0oRySnRs8sDmKixk9z1348rPp9309whv5R44tGlMGd2S8+DZ9oCB8Ghoez6k3OxOIK1TICTU3uCVqse0OGQAqChfv3btWs4//3y6detGz549mTlz5gnbeqN8PcCaNWv45ptvvBJ/IDEOB1kPPkTpL1tJePEFIrt0qVzX8azDREY64aLHYcw38OhOuO1TGPQAJJyryUSp06QtlAASHR3NO++8Q6dOndizZw+9e/fm8ssvp2nTppXbeFq+vjZr1qxh48aNDBs2zCuxBwJjDHv/8Q8KvvuOVk8/TaPBg4+udDpoFFvGd1GNGHrRo/YFqVQQ0xZKAOncuTOdOnUCoE2bNrRs2ZIDBw54vP+2bdu4/PLL6d27N4MHD2br1q0AzJgxg+7du5OSksLFF19McXExf/3rX5k+ffoptW4C1aGpb5P3wQxix42l2c03Hbsyex3RxrApItKe4JQ6A2gLpYpnVz/Lz4d+rnW7im0q+lJO5pzm5/Bo37r/Rbx69WrKysro0KGDx/uMHz+eKVOm0KFDB5YvX87999/PvHnzeOaZZ1i8eDHx8fHk5eURFRXFU089xcaNG3nxxRfrHFsgOjx3HvsnTSJm2DBaPPjgCetN+ndMbRLD9MZRzP/0SgRBRBCrGuIx09a6qsuP377GdULlshOOW2V7p8tQ6jCUOlwUOnLBhDLsg9GESAghElrlFUIIoYRa86EV60Pcy0KrLAuVUEJDjr6HWfuEhYQSKmGEhYQcXRcSRqi1LiwkjNCQEMIkzD0f6t63tCwcAb7Zllb5dXG/H/03pMrX6aiqX0dO+Jod3aJi3jpW5deJarav+DoK5eXuagU/ZG4/2X+JOvHGrReO8nAA0jJ/9cLRvKcirqLyUqLDI2rZ+vRoQglA2dnZ3HbbbUybNo2QEM8akXl5eaxcufKYUisOhwOAgQMHMnr0aG688Uauu+46n8Rsp+K1a9nzxz8SlZJCm4n/RI77mhljeGnHZ7zRvBnRLugR1wNjlXk3Vnl3g8EYc3Temq58rywDT+X8MevM0WMAlDtdFJc7KHU4Kal4dzgpLXdS6nDitErFAwjRgItd+QdAXIALxIXgAjGA071cDOBCqmxTua34oEq01Zh75PsAG+LS+p1458IAG7fHimvMwmvtjeN4Vlyrd2/jouTuPj2VJpQqPG1J+PIur8OHD3PVVVfx97//nf79+3u8nzGGuLi4avtU3njjDVatWsUXX3xBSkoK69ev92bItirbtYvd995HWHw8ia++QkjksZe0jDG8kPYcU8mlX1Ep+aFRPDv4xEHG6sIYQ25ROZm5RWTlFpOZW0xmbpH17p4uLHMes0+jiDASm0WR2CyaxJZR1rR7fszXdyMhZay+85PK4ztdBpcBlzHWC5wuc8w6YwzOinVOFw6Xg3KXk3Kns3La4XJS7nLgrJh3OnGaY6edLhflxoHD5cRp7eM0Tt5c+yEAo7pfC5WJ9+i/rqpjsFT+W7HEPXbLMWnOHE3eldufsKxq4q5ytCrJ/7Ot8wG4pvOldfq++drsrQsAw/BOl9gdyjHmbFsIQKe4Nj4/lyaUAFJWVsaIESMqWxN10axZM1q3bs2sWbMYMWIELpeLDRs2kJKSwo4dO+jfvz/9+vVjzpw5ZGVlERMTw5EjR3z0SfzDmZfH7vG/A6eTsya/Rljz5sesN8YwKW0S725+l5sOHyHSFc7qqNovbhhjOFRYRmZuMVl5JyaLzNxiio5LGDERYSQ2j6ZtbDQDOsaS0NRKHs2iOKtZNI2jwo67LHRUSGjZMfMiQlio/Q9Qvv/L2wD8cfCJBUbtNG/35wD8/dLAajnN3/0FAP+4bKzNkRxrQeZXACQ0bl7LlqdPE0oA+fDDD1myZAk5OTm8/fbbALz99tukpqZ6tP+MGTO45557ePrppykrK2PUqFGkpKTwhz/8gfT0dIwxDB06lO7duxMfH8+kSZPo1asXTzzxBDfccIMPP5n3ucrK2H3//ZRnZdH27alEtG9/zHpjDP9c/U8++PkDRjXuwh/T5zIuvm3lupzCsmpaF0cTR3H5sQmjcWQYic2iSYptyKCOLUhodmwro0lUuN8+u1KBShNKAKgoGT9q1ChGjRrl0T7Vla9PTk5m7ty5J2w7Z86cE5a1aNGCtLS0U4jWfsYYsv/0BMVpP9LmuX8T3bv3MetdxsXfV/6dj7Z+xB3d7uDBTUuQlt1IK+9PQfa5dHnqG0rKXcfs0yQqnMRmUSS3aMjgzi2sFoY7WSQ0i9KEoZQHNKGcAn1C3l4H/vtfDn/xBS3+8AeaXHXVMeucLifPrHiGWdtnMa7HOH7f425k7r/h3NEU/dAR44xg1PntrBaG+5JUQrMoGkdqwlDqdGlCUfVK3iefkPO/12h64w3Ejr/rmHVOl5Mnlz/J5zs+556Ue7gn5R5k9yooL6I0cQDlS4Toppv589V1659SSnlGH2xU9UbBcqsU/cCBtHrqqWM6uR0uB48ve5zPd3zO/an3c2/qve716UsBIY2uYEJp0HCPfR9AqSCnLRRVL7hL0T9ARHIyCf95EQk/eomq3FXOY0seY97OeTxw7gOM7VHlLpuMJdCqO9/tdoA4aRC1z4bolTozaAtFBbzKUvTR0SeUoi93lvPId48wb+c8HunzyLHJxFEKu1dD0gUs23aQ8Mj9SIjDhk+g1JlBE8op2HnbaHbeNtruMM4IrsJCMu+5B2d+Pme99j/CW7euXFfmLOPBxQ+ycNdCHuv7GKO7Hfc9yUwDRwmHW/Vnc/ZhIhpm+Tl6pc4smlACgL/L18+aNYtJkyZ5LX5fqShFX/LzzyS+8DyRXbtWrit1ljJh0QQWZy7myf5PcmuXW088QIa7/2RZ+dkANIjW/hOlfMmWPhQRaQ7MBJKADGCkMSa3mu2cwAZrdpcxZri1vD0wA2gOrAFuM8aUHb9/fePN8vUOh4OwsOq/vSNGBFgNpGoYY9j3f/9nlaL/C40uvLByXbGjmAnfTmBl9kqeGfAM13WqoT5ZxjJo3ZPvdpYRExlGWGSOn6JX6sxkVwvlMWChMaYTsNCar06xMSbVeg2vsvxZ4AVr/1wgsGodnKLTLV8/aNAgnnjiCQYPHszLL7/M7Nmz6devH7169WLo0KHs378fgClTpvDAAw8A7ocpJ0yYwIABA0hOTmbWrFne/2Cn4NDb08h9/wOaj72TZjffXLm8qLyI+xfez8rslfxt4N9qTiblJbB7NSbpApZtP8iADrG+KaColKpk111e1wAXWdPTgMWAR5UZxX2v6BDglir7Pw3873SD2vt//0fpltrL15f87N7Gk36UiC7n0OpPf6pzLKdSvh7cxSWXLFkCQG5uLsOHD0dEeO2113juued49tkTCyPu37+f5cuXs2HDBkaOHGl7C+bw3Hns/9e/iLn8clo+9FDl8sLyQu5dcC9rD6zlnxf8k6uSr6r5IJmrwVnK/ti+ZOUVc/eFyayt/VurlDoNdiWUeGNMNoAxJltEWtawXaSIpAEOYKIx5jMgFsgzxlTcrpMJJNR0IhEZD4wHaNu2rbfi96lTKV9f4eYqf83v2rWLkSNHsnfvXkpLS+ncuXO1+1x77bWICD179iQry96O68pS9D170ubZiZWl6AvKCrhnwT1sOLiBZwc/y7CkWkaazFgGEsKi4g7ATgZ1asHLmlCU8imfJRQRWQC0qmbVE3U4TFtjzB4RSQa+FZENwOFqtqvxWoYx5nXgdYA+ffqc9JqHpy2JipZJu3ff8Wj7ujjV8vUVGjZsWDl933338ac//Ykrr7ySBQsWMHHixGr3iYg4OuiOMfZdFirbvdtdir5lSxL/92plKfrDZYe5e/7dbMnZwr8v/DeXtvOgbHn6UmidyuKMUhKaRpEUG+3j6JVSPksoxpgaf+pFZJ+ItLZaJ62B/TUcY4/1vkNEFgO9gE+ApiISZrVSEoGguH3ndMrXVyc/P5+EhASMMUybNs0LEfrOMaXoX59cWYo+vzSfu+bdxba8bTx/0fNc3Pbi2g9WVgSZP+Dqfy/ff3+QYd1b1Vg6XinlPXZ1ys8BbrembwdmH7+BiDQTkQhrOg4YCGw27j+hFwE3nGz/+qiifH1FyfrU1NQa7+LyxNNPP82IESO48MILiY+P92Kk3uUqKyPz/v9HeWYmia+8XFmKPrckl7Fzx/Jr3q/85+L/eJZMwN1/4iono1EvDpc4GNgxzofRK6Uq2NWHMhH4UETGAruAGwFEpA9wtzFmHNAFmCzu8U5DcPehbLb2fxSYISJ/B34C3vT3B/Amb5WvX7Zs2THz119//TFDAlcYN25c5fR7771XbSz+UlGKvigtjTb//jfRffoAkFOcw7h549h9ZDcvDXmJAQkDPD9o+lKQUBYUJgOZmlCU8hNbEooxJgc4YZxMY0waMM6a/h7oUcP+O4C+vozxZHzRd3KmqixF/8ADNLnafdfWgaIDjJs3juzCbF655BX6te5Xt4NmLIU2vViUXkyX1o2JaxRR+z5KqdOmT8or2+R98ik5/3uNJjdcT+zvxgOwr3Afd869k+zCbF695NW6J5OyQsj6kfK2g/hxZy6DOsb6IHKlVHW02jDuyy7aaVszX9z5Vfj992T/5S80HDCA1n/5CyLC3sK93Dn3Tg6VHGLyZZPp1bJX3Q+8ayW4HPwcmUKZ06WXu5TyozO+hRIZGUlOTo6tt8sGMmMMOTk5RFq38HpDydatZP5+wjGl6LMKsrjjmzvIK8k79WQC7stdIWHMPdyOBqEh9G3f3GtxK6VO7oxvoSQmJpKZmVmnEidnmsjISBITE71yrPJ9+9n9u7sJiYpyl6KPiWH34d2MnTeWwvJC3hj6Bt3iup36CTKWQUJvFqUXcW67pkQ3OOP/iyvlN2f8T1t4eDjtrdtUlW9VLUXf7t13CG/dmp2Hd3Ln3Dspc5YxZegUusR2OfUTlB6BrDUU9fs9mxYf5uGhx1YGSCp7+DQ/gVLqZM74hKL8wzidZD30sLsU/auvENWtGzvydzBu7jicxsmUoVM4u/nZp3eSXavAOFkX2h1A+0+U8rMzvg9Fed+Yb8Yw5psxlfPGGPb94/8oWLyY+D8/QcxFF7E9dztjvhmDy7h46/K3Tj+ZgHu435BwvsprS0xkGD0Smpz+MZVSHtOEonzu0LRp5L7/Ps3vvJPmt9zCL4d+4c65dxIqobw17C06NK1bReUapS/FJPbh218LOD85lrBQ/e+tlD/pT5zyuptf2sTNL20C4PC8eex/9l/EDB1Ky4cfYkvOFsbOG0uD0AZMHTaV5CbJ3jlpyWHIXkt+fD+y8oq5oJNe7lLK37QPRflM8bp17HnEKkX/r2fZdGgz4+ePp1F4I968/E3OijnLeyfbtQKMix+ouf9k5u/O9975lFIn0BaK8onQche777nXXYr+1VfYcOQX7pp3F40bNObtYW97N5kApC+B0Ajm5CTSpkkk7eMa1r6PUsqrNKEorwtxGlpmF2OcTs6aPJkNzl38bv7vaB7ZnLeHvU2bRm28f9KMZZjEPixJP8LAjnFa+UApG+glL+V1TQ+WEFbu4qyXX2J99EHum38f8dHxTBk6hfiGPiijX5wHe9ezr9cE8n8pZ1A96z9ZNeYTu0NQyiu0haK8ynHwIA0LHBQ0CWd9opN7F9xLm4ZtmDpsqm+SCcDO78G4WOHqCsCADvUroSgVLLSForwq76OPEGB9ciivLbyfto3b8sZlbxAb5cOqvxnLICySWftbcU4raBGj5eq9IVBbThpX3fgzLm2hKK8xDge5M2aS0ySEly9z0L5Je94c+qZvkwlAxhKcCX1ZubOQQfp0vFK20YSivObIwm9x7NvHxwOE5oXClKFTaBbZzLcnLToEezeyu0lvd7n6etZ/olQw0YSivCZ3+nSc8bF82xMaJrajSYQfSp/s/B4wLHWcTXio0E/L1StlG48SiogMFJGG1vQoEXleRNr5NjRVn5Rs3UrR6tWs6t+U0LBwmkf66Rd7xlIIj+bj7HjObdtMy9UrZSNPWyj/A4pEJAX4I7AT0IHVVaXcDz6ABuG8mZRBi6gWhIifGr/pSylv05d12cXaf6KUzTz9qXcY95CG1wD/Mcb8B4jxXViqPnEeOUL+7Dns7NuW4kbhtIxq6Z8TF+bA/k382jAVQPtPlLKZpwnliIg8DowCvhSRUCDcd2Gp+iR/1meYoiLeOmcvVyRdQXion/5r7FwGwKLSs4mJDKOnlqtXylaeJpSbgFJgrDFmL5AATPJZVKreMC4Xue+/T0GnNmxpUcqtXW/138nTl0J4Q2buidNy9UoFAI9bKLgvdS0Vkc5AKvCB78JS9UXhihWUZWTwWWop57Y8l26xpzEefF1lLKO49Xlk5Na/citKBSNPE8oSIEJEEoCFwBjgbV8FpeqP3Onv42zSiK+S8ri1ix9bJwUH4MAWfo60+k+0Q14p23maUMQYUwRcB7xkjBkB+PFPURWIyjKzKFi0iNV9m9CiSRuGtB3iv5Nb/SfzizvRukkkyVquXinbeXrTvojI+cCtwFhrWahvQlL1Rd7MGSDCO533cvs5DxEW4v7vNHXYVN+fPH0ppkEMMzNjubirlqtXKhB42kJ5AHgcmGWM2SQiycAi34WlAp2rpIS8jz5mV2prippHc12n6/wbQMZSjsSfR06xS4f7VSpAeNRCMcZ8B3wnIjEi0sgYswP4vW9DU4Hs8Fdf48zL490uxfwm+Xr/lFmpcGQfHNzKxuQrAC1Xr1Sg8LT0Sg8R+QnYCGwWkR9F5JT7UESkuYjMF5Ft1nu1FQRFxCkia63XnCrL3xaR9CrrUk81FlV3xhhyp0+nMLE5685y+LczHtzlVoCvCjpyTqsYLVevVIDw9JLXZOBBY0w7Y0xb4CHgjdM472PAQmNMJ9x3jT1Ww3bFxphU6zX8uHWPVFm39jRiUXVUsn49JZs28Xmqg4EJg0humuzfADKWYiJi+HhPrN7dpVQA8TShNDTGVPaZGGMWA6dzW801wDRrehpw7WkcS/nZoenTcUVF8FXnQv+3TgAylpEbdx4lDrR+l1IBxNOEskNEnhSRJOv1ZyD9NM4bb4zJBrDeayr+FCkiaSKyUkSOTzr/EJH1IvKCiOg1Dz9x5ORw5OtvSDs3hlYt2jMwYaB/AzicDTnb+Sm0B+GhQl8tV69UwPA0odwJtAA+BWZZ02NOtoOILBCRjdW8rqlDfG2NMX2AW4AXRaSDtfxx4BzgPKA58OhJ4hhvJaW0AwcO1OHUqjp5H32MKS/n/W653NrlVv9VFa5g9Z/Mzk+mV9tmNIzQcvVKBQpP7/LKpY53dRljLq1pnYjsE5HWxphsEWkN7K/hGHus9x0ishjoBfxa0boBSkVkKvDwSeJ4HXgdoE+fPqYun0Edyz3E7wyyusRxpJWT4R2O79byg4yluCKa8OX+WCZcqpe7lAokJ00oIvI5UOMv4Wo6yj01B7gdmGi9z67m3M2AImNMqYjEAQOBf1nrKpKR4O5/2XiKcag6OPLttzj27uWDQWFc3/kOosOj/R9E+lL2N++DMz9EO+SVCjC1tVD+7aPzTgQ+FJGxwC7gRgAR6QPcbYwZB3SBI0JaAAAdeklEQVQBJouIC/eluYnGmM3W/tNFpAUgwFrgbh/FqarInf4+xbGN+LFjKX8757f+DyA/E3LT+aHNb4iJCCMlUcvVKxVITppQrAcavc4YkwNcUs3yNGCcNf090KOG/f1YNEoBlG7fTtGqVXx1SRQXt7uENo3a+D+IDHf9rk8OJdG/g5arVyrQeNSHIiIbOPHSVz6QBvzdShAqiOW+/z6u8DC+6l7Gf+24VRggfSnOyGZ8l9eSpwfr5S6lAo2nt8h8DTiB9635m3FfbsrHXcb+N16PTAUMZ0EB+Z/NZm2PaBIT29E7vrc9gWQsIatJb0ye9p8oFYg8TSgDjTFVHzjYICLLjTEDRWSULwJTgSP/s9m4ior4qEcoY7rcak9l39ydkLeLlS2uoVXjSDq00HL1SgUaTy9CNxKRfhUzItIXaGTNOrwelQoYFXW79rZrTG5yLFe0v8KeQKz+k5kHkxjUScvVKxWIPG2hjAPeEpFGuC91HQbGikhD4J++Ck7Zr2jFCsrS0/no6hBuOvt2GoQ2sCeQjKU4IpvzY14rbtPLXUoFJE8fbPwB6CEiTXCP3phXZfWHPolMBYRD09+nNCaCtG7w17NH2hOEMZCxjJ0x50KeMKBjrD1xKKVOytPy9U1E5HnclYEXiMhzVnJRQaw8yz3E77yehks7XklclE0tg9wMyN/N0vJzODs+hpYxkfbEoZQ6KU/7UN4CjgAjrddhwA/jvCo75c6YicHwVYqTW7vadKswVNbvmnkwSe/uUiqAedqH0sEYc32V+WdERMcgCWKu0lJyP/qIDWdH0q5TD7rFnvJ4aqcvYxllkXFsyWvNH3W4X6UClqctlGIRGVQxIyIDgWLfhKQCweGvv8aVl8dnqaX2jHlSwRhIX8r26FTCQkK0XL1SAczTFso9wLSKTnngEHCHr4JS9sud/j458VHkdI1lSFsbK90c2gFH9rCo4QjO1XL1SgU0T+/yWgukiEhja/6wT6NStipev56SDRv4bGgIv+1yC2EhNv4St/pPPs1tz/BeerlLqUBWW/n6B2tYDoAx5nkfxKRsljt9OuURYaxOjeTJTtfZG0z6UkoiW/BrSWsGddLbhZUKZLX96RnjlyhUwHAcOkT+V1+zqCdc1mU4TSJsvDvcev7kl8gUGkWEk5LY1L5YlFK1qq18/TP+CkQFhryPPobycr4+N5RX7eyMB8jZDgV7mdvgOvona7l6pQJdnX9CRWSNLwJR9jMOB7kffMDPyQ1I6jmI5KbJ9gaUvgSAL490ZJA+Ha9UwDuVP/m0Kl+QOrJoEY69e/k81WHvrcIVMpZRFNmSnSaeQfr8iVIB71Ru3/nS61GogJA7/X3ym4aTc25bBiYMrH0HX7L6TzaGp9CqcRQdWjSqfR+llK3q3EIxxvzZF4Eoe5X++itFK1fyZYqT33YfRYjY3F9x4Bco3M9XBR0Z2FHL1StVH3haHPKIiBw+7rVbRGaJiM0X2pU35E5/H2dYCKv6xDC8w3C7w6l8/mRhSWe9XVipesLTS17PA3twDwEsuIcAbgX8grtw5EW+CE75h7OggNzPZrH8HBiaegPR4dF2hwQZSzkS0YrdJS0Z2EH7T5SqDzy9rjHMGDPZGHPEGHPYGPM6cKUxZibQzIfxKT/Inz0bioqZ2yeE33b5rd3hgMsFGctYG9qDzvExtGys5eqVqg88TSguERkpIiHWq+pIS8YXgSn/MMZw6L33SG8TSrt+l5LQKMHukODAz1CUw5dHOjCoYwu7o1FKecjThHIrcBuwH9hnTY8SkSjgfh/FpvygaOVKytMz+KqXCYxbhaGy/2RpeRftP1GqHvG0OOQO4Dc1rF7mvXCUvx2a/j6F0aEcGngOveN72x2OW/oS8iLasK+sJX3ba0JRqr7w9C6vziKyUEQ2WvM9RURvH67nyvfs4ci3C5mX4uKmnrcFxq25LhfsXE4a3ejVtimNtFy9UvWGp5e83gAeB8oBjDHrcd/ppeqx3BkzMcbwQ//mXNH+CrvDcdu/CYpzK58/UUrVH54mlGhjzOrjljm8HYzyH1dpKTkfzuTHjsJl/W6hQWgDu0NyS3f3n6xwduUCLbeiVL3iaUI5KCIdsO7oEpEbgGyfRaV87sg330BePvPPC2Pk2SNr38FfMpaR0yCBIxHx9NRy9UrVK55eoL4PeB04R0SygHTcd36peurAu++SHRvCWRddRVxUgLQEXE7YuYwVrr70T25OuJarV6pe8fQnNguYCvwDmAHMB24/1ZOKSHMRmS8i26z3ah+OFJG2IjJPRLaIyGYRSbKWtxeRVdb+M0UkQK7X1A/FGzZQvnETX58Lt3YbZXc4R+3bCCX5zC/qpP0nStVDniaU2bhvGy7HXYKlACg8jfM+Biw0xnQCFlrz1XkHmGSM6QL0xf0cDMCzwAvW/rnA2NOI5YxzaPp0ShoIeUN60S22m93hHFXRf+Lqpv0nStVDnl7ySjTGDPPiea/haP2vacBi4NGqG4hIVyDMGDMfwBhTYC0XYAhwS5X9nwb+58X4gpbj0CHyv/yS73rAjb1G2x3OsTKWsi/8LKRBKy1Xr1Q95GkL5XsR6eHF88YbY7IBrPeW1WzTGcgTkU9F5CcRmSQioUAskGeMqbjLLBOosV6IiIwXkTQRSTtw4IAXP0L9lPfxJ0i5gzUD4xnSdojd4RzlcmJ2fs8yxzlarl6pesrTFsog4A4RSQdKcVccNsaYnjXtICILcFckPt4TdYjtAqAXsAuYCdwBzKlm2xrriVmFLF8H6NOnzxldd8w4nRyY/i4b2wkXXziasJAAemgwex1SepjFZWdzsfafKFUvefobpc5PvRljLq1pnYjsE5HWxphsEWnN0b6RqjKBn6yyL4jIZ0B/3OXym4pImNVKScTdr6NqUbB4Mew7wLc3RDKx03V2h3Msq37XSldX/qwJRal6yaNLXsaYndW9TuO8czh6l9jtuDv9j/cD0ExEKsrNDgE2G2MMsAi4oZb91XH2v/M2OTFCwrBraRLRxO5wjpWxjKyws2gWn0i8lqtXql6y60b/icBlIrINuMyaR0T6iMgUAGOME3gYWCgiG3BfZnvD2v9R4EER2Y67T+VNP8df75Tu2EHZqjTm9RJu6X6b3eEcy+nA7Pye78rO0duFlarHbLmIbozJAS6pZnkaMK7K/HzghH4a6zJYX1/GGGxypk+nPBSODOtHctMAG7U5ey1SVsAyRxeu14SiVL2ljyKfAZwFheR++gkrzhGu6zvG7nBOZPWfpNGVfslarl6p+koTyhkgf/ZsQopLWT84gYEJA+0O50TpS9kZ2o62Z7XTcvVK1WOaUIKcMYbsd97i11ZwwdA7CZEA+5Y7yzG7VrK47GwG6dPxStVrAfbbRXlb0arVhOzM4ru+0QzveI3d4Zxoz09IeSErnF0ZpP0nStVrmlCCXPY7b3IkClpfcyPR4dF2h3Oi9CUAbAjrRspZWq5eqfpME0oQK8/OpmzRMhalhHBTSoDdKlwhYynbQ5I4J7m9lqtXqp7Tn+AgduCD6RhjKLh6EAmNaix3Zh9HGa5dK1ladrY+f6JUENCEEqRcZWXkzJzBj52EawbdZXc41cv6kRBHCStcOtyvUsFAE0qQOvz114TlF7Llwnb0ju9tdzjVy1iKC2FHdAodW2q5eqXqO73pP0jtfvt19jeH/sPHB2wpeJO+lK20o2en9gEbo1LKc9pCCULFGzYStmUHS/s14ooOV9kdTvUcpZjdq1jm6KL9J0oFCU0oQWj3tNcpCYdWN/6WBqEN7A6neplphDhLWeHqqg80KhUkNKEEGUduLmVzF7KsRyg3BNoQv1VlLMVFCDmxvbVcvVJBQhNKkNk/831Cy10UD7+IuKjA/cvftWMJm00SqZ2S7A5FKeUlmlCCiHE62T/9XTa2Fa667B67w6lZeQlk/sD3zi5abkWpIKIJJYgcXrSIiAP5bBvSgW6x3ewOp2aZqwlxlbHKdKVfcnO7o1FKeYneNhxEdrz1CoUx0OfGe+0O5eTSl+IkhLI2/YiJDLc7GqWUl2gLJUiU7kgncs3PrOzbmCHtL7M7nJNy7FjCRlcS53ZuZ3coSikv0oQSJHZMfRVHCLT67W2EhQRww7OsiJCsH1nh6qa3CysVZDShBAFXYSFlX3zD6q5h/Oa8AK0qXGH3KkJMOT+FdCdVy9UrFVQ0oQSBPZ/OoEGxg5JrLqZJRBO7wzm5jGU4CSE06XwtV69UkAngayPKE8YY9k6byr5WMOzqCXaHU6vS7d+xyZVMn85t7Q5FKeVl+idiPXd45QoaZubw6yVn06FZB7vDObnSAsL3/qTlVpQKUppQ6rmtU17kSCT0uuX/2R1K7XavIsQ42BKRQictV69U0NGEUo+VZWcTvWIDP/ZtysDki+0Op1YmfSnlhNGw40AtV69UENKEUk8ZY1hy01BwQfytowmRwP9WFm9bzFpXMud1PsvuUJRSPhD4v4XUCYzLxe5//JWE/Q4W9xSGDbjd7pBqV3qEyAPrWenqqvW7lApSmlDqGVNezq8PTaDwvRl81SeEBTd3Ijo82u6wardrJSHGya7GvWnVRMvVKxWM9LbhesRVVMQv990FK9bwyZBI1l6eTNOIxnaH5RHHju9wmTCadh5odyhKKR+xJaGISHNgJpAEZAAjjTG51WzXFpgCnAUY4EpjTIaIvA1cCORbm95hjFnr+8jt48zLY8vY25DN25lxTRNGP/Yuf27Wye6wPFay9Ts2mY707ZxodyhKKR+x65LXY8BCY0wnYKE1X513gEnGmC5AX2B/lXWPGGNSrVdQJ5PyvXvZNHIErp+3M/3W1tz31Gd0qkfJhJJ8onM2ssp0pb+Wq1cqaNmVUK4BplnT04Brj99ARLoCYcaY+QDGmAJjTJH/QgwMpTvS2XzjtZTv3cuHvzubRx+ZRauGrewOq252rSQEFwfjtFy9UsHMroQSb4zJBrDeW1azTWcgT0Q+FZGfRGSSiIRWWf8PEVkvIi+ISIQ/gva3ovXr+fmm6ygqzOeLh/ry53tnBH6trmqUbltMqQkn9hztP1EqmPksoYjIAhHZWM3rGg8PEQZcADwMnAckA3dY6x4HzrGWNwcePUkc40UkTUTSDhw4cKofx+/yly1h+223kBtawvdPXcUTt71JZFj9vDuqZNt3rHF14vzOCXaHopTyIZ8lFGPMpcaY7tW8ZgP7RKQ1gPW+v5pDZAI/GWN2GGMcwGfAudaxs41bKTAVd/9KTXG8bozpY4zp06JFC29/TJ848Pln7P7d3WQ1cfLrP8fy4DWTAnuMk5MpziMmbzNp0k3L1SsV5Oy65DUHqHga73ZgdjXb/AA0E5GKLDAE2AyVSQhx1++4Ftjo02j9KGvaFA788XG2toGiFx9n7MUP1+8yJTu/JwRDYZvzaRCmjz0pFczs+rN3IvChiIwFdgE3AohIH+BuY8w4Y4xTRB4GFlqJ40fgDWv/6VaiEWAtcLffP4GXGWPY8fw/KXvjXX7qFEqr5ydxSacr7A7rtBX8sogwE05810F2h6KU8jFbEooxJge4pJrlacC4KvPzgZ7VbDfEpwH6mXE6+fnJR+DTr1mW2oDez03h3ITz7A7LK8p/XcI6V2cGdG5jdyhKKR/TaxA2c5WVseG+O+HTr5k/qBEXvfpx0CQTig7R5PAvrA/vSed4LVevVLCrpz29wcFZUMi6u24h6qetfHVlC27924f17xmTk3BlLCMEQ1nigPrdD6SU8ogmFJs4Dh1i7egbiPw1my9+257xj31QL58xOZm8zYuINBEk9tD+E6XOBJpQbFCWmcn6224g/EA+8+7pxf+79616+4zJSWUsJc3VmQGdW9sdiVLKD7QPxc8KftnChht+g8nNZ/kjlzLh/neDM5kUHqR5wTa2RqXSukmU3dEopfxAWyh+dOiHFez83XhKQhzs+Nut3HP1E8HZtzD1KsoLcggHTJJe7lLqTKEJxU+y53/BgQf/SG6MoeDZh7jtgnG171SP5RSWE2MiSOqh9buUOlNoQvGDHTOnUvTMv9gdH0KjFycyPGW43SH5XFhpLmnmHPp1Cp671pRSJ6d9KD625ZV/UfqXf/FLu3ASpk7hwjMgmeAsI87ksCvmXBpruXqlzhjaQvERYwxr//owkR98xU/douj3vw/o0PJsu8Pyi23ZB+kEhCYPtjsUpZQfaULxAeNw8MODY4mZt5pV/Zsy7L+f0KrxGVB6xFEKGz6mpXMfh00UHVK0/0SpM4kmFC9zlZSw8q6RNPthG8suT2Dkvz4JugcWT1CcC2lTMasmIwV7OeBqw7OOm3klqX4MF6CU8g5NKF5Ufjif1aOvpenPe1l+UxdGP/UBEaFBOZikW+5OWPk/XGumEVJexApSeLVsDLtpyW0RS7VcvVJnGE0oXlK8N5s1o66l8Z7D/HD3QMb8fjKhIaG171gfZf2I+f4l2DwbpwlhtvN83nReRdsufblvQBIxM64hGB+vUUqdnCYUL8jb8QtbRt9MdH4JWx6/jttH/T34Hlh0uWDrNziXv0To7u8ppCHvOq5iVvhVXDKwF2/0b0dCU+uJ+DZBfolPKVUtTSinae/aley+6y5CHQ72TryHkVf93u6QvKu8GNbNoHzZfwnP28E+4phSfhsb4n/DyAFdmZPShsjwIG2JKaXqRBPKachY/CWHJjxCaQTw8tNcMfAmu0PynsKDuFa/gWPl6zQoPcQWV3umuH6PdB3O6IEdeLJts+BrhSmlTosmlFO0adZUnE/+i9ymocRN/i8p3YJkEMmD2ylb/hIh6z4gzFXKEmcvPmzwAF0GDOOJ/u2Ib+xBIcsxX/o+TqVUwNGEcgrS3phI1PPTyEpsQOc3p9GhbardIZ0eY2DXSgoWv0B0+jyMCeND5yBWtLyJSwcP5uXurfWOLaVUrTSh1IExhuUTHyJ22tdsPbsh/d/6hPjYdnaHdepcTpybP6fg2+dpcmgd5aYRr7pGsL/Lbdww+FxuSWxqd4RKqXpEE4qHjMvF4j/eTqsv0tjcO47LXp9D44bN7A7r1JQWULz6HcqXv0zjkiwOueKZ3GA8TfqP5ub+nYlrFMTPziilfEYTigecZaV8e/f1JH7/KxuGtGP4f2YRGV4PB406speD375E9PppRDuPsMnVmUWxT9Lt4pt5sFsbwkL1spZS6tRpQqmFy+ViwegraLs2m003pHLdM+8SFlq/vmyO7E3s/ebfxO/8nObGwXzOY0fHMVx06dU80rqx3eEppYJE/frNaIOQkBC4dBDb++Vx/QP/qT+3yhpD/uYF5C18gXaHlhNrGjAn7FLKz/sdwwYP5PLoBnZHqJQKMppQPHD5uL/aHYLnnOXsWvoeoStfIaFkG2WmCR81uZ0WF9/LtSlnExpSTxKiUqre0YQSJMoKctn69Su02jKVtq6D/GoS+KzdY/QYNo4b22jVX6WU72lCqecOZv1KxpfPcc6eT+lOMWtCe/BT6lP0u/xmro3Su7WUUv6jCcUDq166nfjcH+0O4wQCtHHuIRVDWqMLCb9gAr36XkSIXtZSStlAE4oHXI0TOFR6yO4wqpXdaDAJQyfQP/nMGF5YKRW4NKF44Pzb/8/uEJRSKuDZ8iSbiDQXkfkiss16P+GRcxG5WETWVnmViMi11rr2IrLK2n+miOg9sEopZTO7Ho1+DFhojOkELLTmj2GMWWSMSTXGpAJDgCJgnrX6WeAFa/9cYKx/wlZKKVUTuxLKNcA0a3oacG0t298AfG2MKRL3k4VDgI/rsL9SSikfsyuhxBtjsgGs95a1bH8z8IE1HQvkGWMc1nwmkFDTjiIyXkTSRCTtwIEDpxm2UkqpmvisU15EFgCtqln1RB2P0xroAcytWFTNZqam/Y0xrwOvA/Tp06fG7ZRSSp0enyUUY8ylNa0TkX0i0toYk20ljP0nOdRIYJYxptyaPwg0FZEwq5WSCOzxWuBKKaVOiV2XvOYAt1vTtwOzT7Ltbzl6uQtjjAEW4e5X8WR/pZRSfmBXQpkIXCYi24DLrHlEpI+ITKnYSESSgLOA747b/1HgQRHZjrtP5U0/xKyUUuokxP0H/5lBRA4AO09x9zjcl9sCjcZVNxpX3WhcdROscbUzxtRaZfaMSiinQ0TSjDF97I7jeBpX3WhcdaNx1c2ZHpeO+aqUUsorNKEopZTyCk0onnvd7gBqoHHVjcZVNxpX3ZzRcWkfilJKKa/QFopSSimv0IRyCkTkYRExIhJndywAIvI3EVlvlfmfJyJt7I4JQEQmicjPVmyzRKSp3TEBiMiNIrJJRFwiYvsdOSIyTER+EZHtInJC5W07iMhbIrJfRDbaHUtVInKWiCwSkS3W93CC3TEBiEikiKwWkXVWXM/YHVNVIhIqIj+JyBe+PI8mlDoSkbNwP4y5y+5YqphkjOlplfr/AnjK7oAs84HuxpiewFbgcZvjqbARuA5YYncgIhIKvAJcAXQFfisiXe2NCoC3gWF2B1ENB/CQMaYL0B+4L0C+XqXAEGNMCpAKDBOR/jbHVNUEYIuvT6IJpe5eAP7ISQpS+psx5nCV2YYESGzGmHlVqkKvxF13zXbGmC3GmF/sjsPSF9hujNlhjCkDZuAe3sFWxpglQMCNe22MyTbGrLGmj+D+JVljtXF/MW4F1my49QqIn0MRSQSuAqbUtu3p0oRSByIyHMgyxqyzO5bjicg/RGQ3cCuB00Kp6k7ga7uDCEAJwO4q8ycdjkEdZZVm6gWssjcSN+uy0lrcxW7nG2MCIi7gRdx/BLt8fSIdU/44tZTd/xMw1L8RuZ0sLmPMbGPME8ATIvI4cD/wl0CIy9rmCdyXKqb7IyZP4woQdRqOQbmJSCPgE+CB41rotjHGOIFUq69wloh0N8bY2gclIlcD+40xP4rIRb4+nyaU49RUdl9EegDtgXXuQSNJBNaISF9jzF674qrG+8CX+Cmh1BaXiNwOXA1cYvx4j3odvl52y8RdALWCDsdQCxEJx51MphtjPrU7nuMZY/JEZDHuPii7b2oYCAwXkSuBSKCxiLxnjBnli5PpJS8PGWM2GGNaGmOSjDFJuH8RnOuPZFIbEelUZXY48LNdsVQlIsNwV4YebowpsjueAPUD0ElE2otIA9yjk86xOaaAZQ0B/iawxRjzvN3xVBCRFhV3MYpIFHApAfBzaIx53BiTaP3Ouhn41lfJBDShBIuJIrJRRNbjviQXELdSAi8DMcB865bm1+wOCEBERohIJnA+8KWIzK1tH1+xblq4H/eIpFuAD40xm+yKp4KIfACsAM4WkUwRGWt3TJaBwG3AEOv/1Frrr2+7tQYWWT+DP+DuQ/HpLbqBSJ+UV0op5RXaQlFKKeUVmlCUUkp5hSYUpZRSXqEJRSmllFdoQlFKKeUVmlCU8iIRKah9q5Pu/7GIJFvTjURksoj8alWwXSIi/USkgTWtDyargKIJRakAISLdgFBjzA5r0RTcBRo7GWO6AXcAcVYRyYXATbYEqlQNNKEo5QPiNsl64HSDiNxkLQ8RkVetFscXIvKViNxg7XYrUFH/rAPQD/izMcYFYFUk/tLa9jNre6UChjaZlfKN63CPi5ECxAE/iMgS3E96JwE9gJa4n45/y9pnIPCBNd0NWGsVHKzORuA8n0Su1CnSFopSvjEI+MAY4zTG7AO+w50ABgEfGWNcVh24RVX2aQ0c8OTgVqIpE5EYL8et1CnThKKUb1RXlv5kywGKcVeEBdgEpIjIyX5GI4CSU4hNKZ/QhKKUbywBbrIGXWoBDAZWA8uA662+lHjgoir7bAE6AhhjfgXSgGesCruISCcRucaajgUOGGPK/fWBlKqNJhSlfGMWsB5YB3wL/NG6xPUJ7qEPNgKTcY82mG/t8yXHJphxuAcJ2y4iG4A3ODpWysXAV779CErVjVYbVsrPRKSRMabAamWsBgYaY/Za42gssuZr6oyvOManwOPGmF/8ELJSHtG7vJTyvy+swZgaAH+rGKTNGFMsIn/BPab8rpp2tgbi+kyTiQo02kJRSinlFdqHopRSyis0oSillPIKTShKKaW8QhOKUkopr9CEopRSyis0oSillPKK/w/HnGUQfC7KEwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fd5b4a7ebe0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#pd.DataFrame(grid.cv_results_).to_csv('LogisticGridSearchCV_Otto.csv')\n",
    "#cvresult = pd.DataFrame.from_csv('LogisticGridSearchCV_Otto.csv')\n",
    "#test_means = cv_results['mean_test_score']\n",
    "#test_stds = cv_results['std_test_score'] \n",
    "#train_means = cvresult['mean_train_score']\n",
    "#train_stds = cvresult['std_train_score'] \n",
    "\n",
    "\n",
    "# plot CV误差曲线\n",
    "test_means = grid.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid.cv_results_[ 'std_test_score' ]\n",
    "train_means = grid.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = grid.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "\n",
    "# plot results\n",
    "n_Cs = len(Cs)\n",
    "number_penaltys = len(penaltys)\n",
    "test_scores = np.array(test_means).reshape(n_Cs,number_penaltys)\n",
    "train_scores = np.array(train_means).reshape(n_Cs,number_penaltys)\n",
    "test_stds = np.array(test_stds).reshape(n_Cs,number_penaltys)\n",
    "train_stds = np.array(train_stds).reshape(n_Cs,number_penaltys)\n",
    "\n",
    "x_axis = np.log10(Cs)\n",
    "for i, value in enumerate(penaltys):\n",
    "    #pyplot.plot(log(Cs), test_scores[i], label= 'penalty:'   + str(value))\n",
    "    pyplot.errorbar(x_axis, test_scores[:,i], yerr=test_stds[:,i] ,label = penaltys[i] +' Test')\n",
    "    pyplot.errorbar(x_axis, train_scores[:,i], yerr=train_stds[:,i] ,label = penaltys[i] +' Train')\n",
    "    \n",
    "pyplot.legend()\n",
    "pyplot.xlabel( 'log(C)' )                                                                                                      \n",
    "pyplot.ylabel( 'neg-logloss' )\n",
    "pyplot.savefig('LogisticGridSearchCV_C.png' )\n",
    "\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上图给出了L1正则和L2正则下、不同正则参数C对应的模型在训练集上测试集上的正确率（score）。\n",
    "可以看出在训练集上C越大（正则越少）的模型性能越好；但在测试集上当C=0.1时性能最好（L1正则和L2正则均是）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "用LogisticRegressionCV实现正则化的 Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "L1正则"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegressionCV(Cs=[1, 10, 100, 1000], class_weight=None, cv=5,\n",
       "           dual=False, fit_intercept=True, intercept_scaling=1.0,\n",
       "           max_iter=100, multi_class='ovr', n_jobs=1, penalty='l1',\n",
       "           random_state=None, refit=True, scoring='neg_log_loss',\n",
       "           solver='liblinear', tol=0.0001, verbose=0)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegressionCV\n",
    "\n",
    "Cs = [1, 10,100,1000]\n",
    "\n",
    "# 大量样本（6W+）、高维度（93），L1正则 --> 可选用saga优化求解器(0.19版本新功能)\n",
    "# LogisticRegressionCV比GridSearchCV快\n",
    "lrcv_L1 = LogisticRegressionCV(Cs=Cs, cv = 5, scoring='neg_log_loss', penalty='l1', solver='liblinear')\n",
    "lrcv_L1.fit(X_train, y_train)    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{1: array([[-0.46345936, -0.46108523, -0.46086542, -0.460844  ],\n",
       "        [-0.46426116, -0.4646347 , -0.46467172, -0.46469632],\n",
       "        [-0.56469589, -0.56847827, -0.56880508, -0.56883135],\n",
       "        [-0.44366062, -0.44397851, -0.44402747, -0.44403485],\n",
       "        [-0.46581604, -0.46606067, -0.46612281, -0.46613105]])}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lrcv_L1.scores_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.42001018,  1.11874756, -0.23277336,  0.03899002, -0.10286188,\n",
       "         0.72279325,  0.20819848,  0.13338932]])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lrcv_L1.coef_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "惩罚不够，没有稀疏系数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "L2正则"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegressionCV(Cs=[1, 10, 100, 1000], class_weight=None, cv=5,\n",
       "           dual=False, fit_intercept=True, intercept_scaling=1.0,\n",
       "           max_iter=100, multi_class='ovr', n_jobs=1, penalty='l2',\n",
       "           random_state=None, refit=True, scoring='neg_log_loss',\n",
       "           solver='liblinear', tol=0.0001, verbose=0)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegressionCV\n",
    "\n",
    "Cs = [1, 10,100,1000]\n",
    "\n",
    "# 大量样本（6W+）、高维度（93），L2正则 --> 缺省用lbfgs，为了和GridSeachCV比较，也用liblinear\n",
    "\n",
    "lr_cv_L2 = LogisticRegressionCV(Cs=Cs, cv = 5, scoring='neg_log_loss', penalty='l2', solver='liblinear')\n",
    "lr_cv_L2.fit(X_train, y_train)    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{1: array([[-0.46129644, -0.46088502, -0.4608487 , -0.46084513],\n",
       "        [-0.46510588, -0.46473286, -0.46469927, -0.46469595],\n",
       "        [-0.56426612, -0.56834734, -0.56879096, -0.5688357 ],\n",
       "        [-0.44377717, -0.44400066, -0.4440332 , -0.44403656],\n",
       "        [-0.46617809, -0.46612355, -0.4661268 , -0.46612722]])}"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_cv_L2.scores_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SVM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "default SVC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "from sklearn.metrics import confusion_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.svm import LinearSVC\n",
    "\n",
    "SVC1 = LinearSVC().fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "classification_report:\n",
      "\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "          0       0.75      0.89      0.81        99\n",
      "          1       0.69      0.45      0.55        55\n",
      "\n",
      "avg / total       0.73      0.73      0.72       154\n",
      "\n",
      "confusion_matrix:\n",
      "\n",
      " [[88 11]\n",
      " [30 25]]\n"
     ]
    }
   ],
   "source": [
    "#在校验集上测试，估计模型性能\n",
    "y_predict = SVC1.predict(X_test)#输出的是函数的预测值\n",
    "\n",
    "print('classification_report:\\n\\n',classification_report(y_test, y_predict))#打印出来实验报告\n",
    "print('confusion_matrix:\\n\\n',confusion_matrix(y_test, y_predict))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从中可以看出平均精度为0.73，混淆矩阵中预测对的数目于误判的数目"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fit_grid_point_Linear(C, X_train, y_train, X_test, y_test):\n",
    "    \n",
    "    # 在训练集是那个利用SVC训练\n",
    "    SVC2 =  LinearSVC( C = C)\n",
    "    SVC2 = SVC2.fit(X_train, y_train)\n",
    "    \n",
    "    # 在校验集上返回accuracy\n",
    "    accuracy = SVC2.score(X_test, y_test)\n",
    "    \n",
    "    print(\"accuracy: {}\".format(accuracy))\n",
    "    return accuracy "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No handles with labels found to put in legend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy: 0.7142857142857143\n",
      "accuracy: 0.7142857142857143\n",
      "accuracy: 0.7207792207792207\n",
      "accuracy: 0.7402597402597403\n",
      "accuracy: 0.7337662337662337\n",
      "accuracy: 0.7402597402597403\n",
      "accuracy: 0.7597402597402597\n",
      "accuracy: 0.7272727272727273\n",
      "accuracy: 0.7077922077922078\n",
      "accuracy: 0.6428571428571429\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xl4VOXZx/HvzW4FRYFaa1iiRStoRUnxVYqyuLA1U8VXwVerbZXailbrUi21WuyiUrcqVtHaVqpSJQooaERBXKkEcWFVxIXgAiKKKMh2v388kxpDwgwhZ84sv8915SJz5szMPZrkN+d5zrkfc3dERES2pVHcBYiISPZTWIiISEoKCxERSUlhISIiKSksREQkJYWFiIikpLAQEZGUFBYiIpKSwkJERFJqEncBDaVt27beqVOnuMsQEckpc+bM+dDd26XaL2/ColOnTlRUVMRdhohITjGzt9PZT8NQIiKSksJCRERSUliIiEhKeTNnISJS6DZu3EhlZSXr16/f6r4WLVpQVFRE06ZN6/XckYaFmfUHbgQaA3e4+1U17r8e6JO8+TXg6+7eOnlfB+AOoD3gwEB3fyvKekVEclllZSWtWrWiU6dOmNl/t7s7q1atorKykuLi4no9d2RhYWaNgTHA0UAlMNvMJrv7gqp93P38avufAxxc7SnuAv7g7tPMrCWwJapaRUTywfr167cKCgAzo02bNqxcubLezx3lnEUPYIm7L3X3DcB4ILGN/YcB9wKYWRegibtPA3D3te7+eYS1iojkhZpBkWp7uqIMi72AZdVuVya3bcXMOgLFwPTkpn2Bj83sATOba2ajk0cqIhKT9evhjjtgzZq4K5E4RBkWtcVYXQt+DwUmuPvm5O0mQC/gQuC7wN7A6Vu9gNlwM6sws4odObwSkW37/HMoLYUzz4Tzz0+9v+SfKMOikjA5XaUIeLeOfYeSHIKq9ti5ySGsTcBE4JCaD3L3se5e4u4l7dqlvFpdROph7VoYNAgefxyOPBLuvBOeeSbuqqQu7rV/Jq9re7qiDIvZQGczKzazZoRAmFxzJzPbD9gNeL7GY3czs6oE6AssqPlYEYnWmjXQvz88/TT8618wZQp07AhnnQUbN8ZdndTUokULVq1atVUwVJ0N1aJFi3o/d2RnQ7n7JjMbAZQTTp29093nm9kooMLdq4JjGDDeq707d99sZhcCT1iYlZkD3B5VrSKytY8/DkExZw6MHw8nnBC233RTGJK6/nq4+OJ4a5SvKioqorKystaznqqus6gv29FDk2xRUlLiaiQo0jA++giOPhpefRXuvx8SNc5jPO44eOwxWLAgHGlI7jKzOe5ekmo/tfsQka9YuRL69IH582HixK2DAuDGG8O/556b2dokPgoLEfmv99+H3r3h9dfhoYdg4MDa9+vQAX73O5g8GSZNymiJEhOFhYgAsHx5ONvp7bdh6tQwDLUtv/gFHHggnHNOOGNK8pvCQkR4550QFO+9B+Xl4egilaZN4a9/hWXLYNSoyEuUmCksRArcm2/CEUfAhx/CtGnQs2f6j+3ZE844A667LkyGS/5SWIgUsNdfD0Hx6acwfToceuj2P8dVV8Fuu4VrL7ao3WfeUliIFKiFC8PQ0/r1ISgO2apHQnratIHRo+G55+Dvf2/YGiV7KCxECtC8eWFeYssWePJJOOigHXu+004LRygXXxxOvZX8o7AQKTBz54agaNIEZs6Erl13/DnNwmT3mjW6qjtfKSxECsjs2dC3L+y8Mzz1FOy3X8M9d5cucNFF8I9/hOeW/KKwECkQzz8PRx0VJqNnzoR99mn41/jNb6BTJ/jZz2DDhoZ/fomPwkKkADz1FBxzDOyxRwiKTp2ieZ2vfQ1uvjn0jLruumheQ+KhsBDJc088AQMGQFFRCIr27VM/ZkcMGgTHHx8u1HvrrWhfSzJHYSGSx8rLYfBg2HvvcNbTnntm5nVvuAEaNYIRIyBPGlsXPIWFSJ56+OGw7sS3vw0zZoQhqExp3z4cWUyZEjrXSu5TWIjkoQcfDENB3/lOGIZq2zbzNZx7brh+49xz1WgwHygsRPLMv/8N//u/UFIS1s3effd46mjSJFx7UVkJV1wRTw3ScBQWInlk3Dg4+eTQ4K+8HHbdNd56DjsMhg8PcxivvBJvLbJjFBYieeLOO0Pbjd69w3oUrVrFXVHwpz+Foxs1GsxtCguRPHDrrfCTn4RrKR5+OFyhnS123x3+/OdwUeDf/hZ3NVJfCguRHPeXv4QrpgcPDmce7bRT3BVt7dRTQ4fbX/0KVqyIuxqpD4WFSA4bPTosb3r88VBWBi1axF1R7aoaDa5dq0aDuUphIZKjfv/78Id36FAYPx6aNYu7om3bf//QaPCf/wwXCEpuUViI5Bh3+O1v4bLLwvDOuHFhPexcMHIkFBfDz3+uRoO5RmEhkkPc4dJL4cor4cc/DivTNWkSd1Xpq2o0uHAhXHtt3NXI9og0LMysv5ktNrMlZnZJLfdfb2YvJb9eM7OPa9y/i5ktN7Obo6xTJBe4wy9/CVdfHSa0b78dGjeOu6rtN3AgDBkS2oEsXRp3NZKuyMLCzBoDY4ABQBdgmJl1qb6Pu5/v7t3cvRtwE/BAjae5EpgZVY0iuWLLltCU74YbwoT2mDGhUV+uuuGGcER0zjlqNJgrovxx6wEscfel7r4BGA8ktrH/MODeqhtm1h3YA3gswholSz35JFxzDSxaFHcl8XKHF18McxO33BImtK+/PpxdlMuKisJQ2tSpoY+VZL8ow2IvYFm125XJbVsxs45AMTA9ebsRcC1wUYT1SZaaPBmOPTack7///mGN6MsvD+0iCuFT6JYtMGtWOHNon32ge/fQ7+mKK+Cqq3I/KKqMGAHduoVGg59+Gnc1kkqUYVHbj3Rdv+pDgQnuvjl5++fAVHdfVsf+4QXMhptZhZlVrFy5cgdKlWxRVhbGs7t1g3nzwgVn7dqF00QPOgj23RcuuSSsJZ1PwbF5c1iY6NxzoUOH0FPpxhtDe/E77oD33w+BmS9BAWEY6tZb4d13w3uTLOfukXwBhwHl1W5fClxax75zgcOr3b4beAd4C/gQWANcta3X6969u0tuu+ce98aN3Q8/3P2TT7563/vvu992m/vRR7s3aeIO7h06uJ93nvszz7hv3hxPzTtiwwb3xx5zHz7c/etfD++pRQv3H/zAfdw499Wr464wM846y71RI/e5c+OupDABFZ7G33TziD6emVkT4DWgH7AcmA2c7O7za+y3H1AOFHstxZjZ6UCJu4/Y1uuVlJR4RUVFA1UvmfbPf4ZTQXv1Cr2NWrase9+PPgpDVWVl8Nhj4Xz9PfeE444LRyVHHJG9p5N+8QVMmxZqnzQJVq8OfZwGDQq1Dxy47feej1avDkdQxcXw3HO5PXGfi8xsjruXpNwxnUSp7xcwkBAYbwAjk9tGAaXV9rmCbRw1AKcDN6d6LR1Z5K7bb3c3cz/qKPfPPtu+x37ySTgiGTLEfaedwqfztm3df/IT96lT3b/4Ipqat8dnn7lPmOA+bJh7q1ahxl13dT/1VPeJE90//zzuCuM3blz473LrrXFXUniI+8gi03RkkZtuuQXOPhv694cHHtixJniffw6PPgoTJoSjk08/Des5lJaGT+3HHJO5Jntr1oQlRcvKwhk/69aF1ep+8INQS9++2d+eI5PcoV8/mDs3nAGXySVgC126RxYKC4nNDTfA+eeHP+b33QfNmzfcc69fH1aJy+RwTy4Pj2WDRYvCMrBDh8Jdd8VdTeFQWEhWu/rqcFbTkCFwzz3RfsreuDFct1FWFs7pX7EidGc99tjw+t//PrRuXb/n/uCD0Ba8rAxmzIBNm8LZTEOGhK/DDtMY/Pa47LJw5tv06dCnT9zVFAaFhWStK68MjfCGDQufIDP5aXvzZnj22fDHvawMli8PTfiOOir8cU8kwnDRtixfHobMysrg6afDdRHf+haccEJ4ju7d8+sU10xatw4OOCD8P3n55YY92pTaKSwk61R1S/397+GHPwzLgMbZ22jLFnjhhS+D4803Qz1HHhn+6B93XBhGgnBf1X6zZoVtXbt+eQRx4IEKiIby6KMwYED4ORk5Mu5q8p/CQrKKe7gie/RoOOMMuO227BqecYeXXvoyEBYtCn/8Dz88fNp98cWw3yGHfBkQ++0Xb8357MQT4aGHwoWZ++wTdzX5TWEhWcM9TGTfeGNYx+Cmm7IrKGqzYEEIjYkTw1DI8ceHgCgujruywrB8eWj10rNnOJtMR23RUVhIVtiyJZwae+utcN55cN11+sWX9Nx4Y/iZuf/+MB8k0Ug3LLL8853kss2b4cwzQ1D86lcKCtk+Z58NBx8cWrKvWRN3NaKwkEhs2gSnnx4msX/7W/jTnxQUsn2qGg2+9174GZJ4KSykwW3cCKecAv/6Vzij5Xe/U1BI/fToAWedFea55s6Nu5rCprCQBrVhA5x0Ulh/YfRonfooO+6PfwzXvvz0p2FoU+KhsJAGs359OGPowQfD5OSFF8ZdkeSD1q3D6oCzZ8PYsXFXU7gUFtIg1q0LTfIefhj++tewiI9IQxk2LDQavPTS0GJFMk9hITvss89g8ODQPO9vfwtjzCINyQzGjAkfSi64IO5qCpPCQnbIp5+G1gxPPhn6PP34x3FXJPlqv/1C88m774Ynnoi7msKjsJB6++ST0Ln1uedC59hTTom7Isl3l14a2n/8/Ofhg4pkjsJC6mX16tCptaIirEVx0klxVySFoEWL0FfsjTfCB5VPPom7osKhsJDt9uGHYaW3V14JrbqPPz7uiqSQ9OsXTs2ePRuOPjp8cJHoKSxku3zwQViUZtGisCrc4MFxVySFaMiQ0Ojx5ZdDeHz4YdwV5T+FhaTt3Xehd29YujSsL33ssXFXJIWstDQsmbtgQTjSXbEi7orym8JC0rJsWVgUqLIyLE7Tt2/cFYlA//7hg8uSJeGDzHvvxV1R/lJYSEpvvRWCYsWKcC1Fr15xVyTypX794JFH4J13vvxAIw1PYSHb9MYb4Rdw9Wp4/HE47LC4KxLZ2pFHhg8yH3wQvn/77bgryj8KC6nT4sVwxBHhCu0ZM+C73427IpG6HX44TJsGH30Ufm7feCPuivKLwkJqNX9++IS2aVO4Ortbt7grEkmtR49wdffateHnd/HiuCvKH5GGhZn1N7PFZrbEzC6p5f7rzeyl5NdrZvZxcns3M3vezOab2Stmpku+Mujll8NkYaNGISgOOCDuikTSd8gh4ed2w4bwc7xgQdwV5YfIwsLMGgNjgAFAF2CYmXWpvo+7n+/u3dy9G3AT8EDyrs+BH7p7V6A/cIOZtY6qVglWrgwtoPv2DVfKzpwJ++8fd1Ui2+/AA0NgQAiMV16Js5r8EOWRRQ9gibsvdfcNwHggsY39hwH3Arj7a+7+evL7d4EVQLsIay1Yy5fDzTeHC+2+8Y2wwMw3vxmConPnuKsTqb8uXcLPcbNm4ef7xRfjrii3RRkWewHLqt2uTG7bipl1BIqB6bXc1wNoBmw1XWVmw82swswqVq5c2SBFF4K33oJrrw0TgkVFcM454bTYkSPhpZfCp7C99467SpEdt+++8NRT0KpVOMX2hRfirih3NYnwuWtbddnr2HcoMMHdv7JoopntCYwDTnP3LVs9mftYYCxASUlJXc8twGuvhfYIZWUwZ07Y1q1bWCN7yBD49rfjrU8kKnvvHY4w+vYNzS8feQR69oy7qtwTZVhUAu2r3S4C3q1j36HA2dU3mNkuwBTgN+4+K5IK85g7zJv3ZUDMmxe2H3ooXHNNCAgdPUih6NgxBEa/fqFNzZQp4WwpSV+UYTEb6GxmxcByQiCcXHMnM9sP2A14vtq2ZsCDwF3ufn+ENeYV9zAuW1YGEybA66+HFcZ69QprYh93HLRvn/p5RPJRUVGY9D7qqLBg1+TJ4XtJT2Rh4e6bzGwEUA40Bu509/lmNgqocPfJyV2HAePdvfow0onAEUAbMzs9ue10d38pqnpz1ZYtMGtWCIgHHgjzEY0bhwm9Cy4I62LvsUfcVYpkhz33DBeYHnVU6Jj84IMhOCQ1++rf6NxVUlLiFRUVcZeREZs2wdNPh4B48MHQDbZZs9Dbf8iQ0I2zTZu4qxTJXqtWwTHHhOHZ++8PvzOFyszmuHtJqv2iHIaSBrRxI0yfHgJi4sRwTcROO4VPRUOGwKBBsOuucVcpkhvatAlXeh97bPj9ufdeOOGEuKvKbgqLLLZ+fWiOVlYWxlc//hhatgyHz0OGhKDYeee4qxTJTa1bh15SAwbA0KEwbhwMGxZ3VdlLYZFlPvssnNo3YUI4Y2Pt2vBDnUiEgDj66HB1tYjsuF12gfLy8AHslFNCi5DTTou7quxU8GGxZg2MGBF3FcHq1eHQeN06aNcOTj45BESfPtC0adzVieSnli1h6tTwgexHPwpDvmecEXdV2afgw2LjRnjmmbirCJo3h5/8JAREr17hrCYRid7XvgYPPQTHHw9nnglffAFnn536cYWk4MOiTZuwprSIFLYWLcLZhSedFEYbNmyA88+Pu6rskVZvKDMrM7NBZqb1L0QkbzVvHk6lPeEE+OUv4eqr464oe6R7ZPFX4EfAX8zsfuAf7r4ourJEROLRtGk4lbZpU7jkknCEcdllcVcVv7TCwt0fBx43s10JV1xPM7NlwO3Av9x9Y4Q1iohkVJMm4VTaZs3gt78NcxhXXhna5xSqtOcszKwNcApwKjAXuBv4HnAa0DuK4kRE4tK4Mdx5ZwiMP/whBMY11xRuYKQVFmb2APBtQrvw77v7e8m7/m1mhdFjQ0QKTqNGcOutITD+/OcwJHXDDYUZGOkeWdzs7lstTASQTk8REZFc1agR3HRTmPy+7roQGGPGhO2FJN2w2N/MXnT3jwHMbDdgmLvfEl1pIiLZwSwcWTRrBlddFQJj7NjCuhYq3Ww8syooANx9NXBmNCWJiGQfM/jjH+Hyy8Ncxumnhw7QhSLdI4tGZmZVa06YWWPCutgiIgXDDK64IhxhjBwZbt91V9xVZUa6RxblwH1m1s/M+gL3Ao9GV5aISPb69a/hoovC6bVvvRV3NZmRblj8CpgO/IywVvYTwMVRFSUiku2GDw//Tp687f3yhVbKExGpp65d4RvfCN2ic1W6K+Wl2xuqs5lNMLMFZra06mvHyxQRyV2JBMycGZYXyHfpDkP9ndAfahPQB7iLcIGeiEjBKi2FzZvDgmX5Lt2w2MndnyAMW73t7lcAfaMrS0Qk+/XoEYahJk2Ku5LopXvq7Ppke/LXzWwEsBz4enRliYhkv0aN4Pvfh/HjQ++o5s3jrig66R5ZnAd8DTgX6E5oKKiVakWk4CUS8Omn8OSTcVcSrZRhkbwA70R3X+vule7+I3cf4u6zMlCfiEhW69cPdt45/4eiUoaFu28GupsVYp9FEZFta9ECjj02XG+RJ1ci1CrdYai5wCQzO9XMjq/6SvUgM+tvZovNbImZXVLL/deb2UvJr9fM7ONq951mZq8nvzTkJSJZq7QUli+HF1+Mu5LopDvBvTuwiq+eAeXAA3U9IDl8NQY4GqgEZpvZZHdf8N8ncD+/2v7nAAcnv98duBwoSb7OnORjC+BsZhHJNYMGhcnuSZOge/e4q4lGusuq/qgez90DWOLuSwHMbDyQABbUsf8wQkAAHAtMc/ePko+dBvQn9KQSEckqbdvC974XwmLUqLiriUa6K+X9nfAJ/yvc/cfbeNhewLJqtyuBQ+t4/o5AMaH/VF2P3auWxw0HhgN06NBhG6WIiEQrkYALLoA334Ti4riraXjpzlk8DExJfj0B7AKsTfGY2ibE65r+GQpMSE6mp/1Ydx/r7iXuXtKuXbsU5YiIRCeRCP/ma2PBtMLC3cuqfd0NnAgckOJhlUD7areLgHfr2HcoXx1i2p7HiojEbp99oEuXAg+LWnQGUo37zAY6m1mxmTUjBMJW/xnNbD9gN+D5apvLgWPMbLfkEq7HJLeJiGStfG4smG7X2U/NbE3VF/AQYY2LOrn7JmAE4Y/8QuA+d59vZqPMrLTarsOA8V6tV3pyYvtKQuDMBkZVTXaLiGSrRCI0Fpw6Ne5KGp7WsxARaSBbtkBRUTgz6r774q4mPQ29nsVxZrZrtdutzewHO1KgiEi+qWos+MgjobFgPkl3zuJyd/+k6oa7f8yX10SIiEhSIgFr18KMGXFX0rDSDYva9kv36m8RkYLRt29oLJhvZ0WlGxYVZnadme1jZnub2fXAnCgLExHJRfnaWDDdsDgH2AD8G7gPWAecHVVRIiK5LJEIjQXn5NFH6nR7Q30GbNU1VkREtjZoEDRuHHpFlaQ8zyg3pHs21DQza13t9m5mpovkRERq0abNl40F80W6w1Btk2dAAZBsFa41uEVE6pBIwKuvhsaC+SDdsNhiZv9t72Fmnai7KaCISMErTfapyJezotINi5HAM2Y2zszGATOBS6MrS0Qkt+2zD3Ttmj9DUel2nX2UsGrdYsIZURcQzogSEZE6JBLw1FPwUR50tkt3gvsMwjoWFyS/xgFXRFeWiEjuy6fGgukOQ/0C+C7wtrv3IayVvTKyqkRE8kBJCey5Z34MRaUbFuvdfT2AmTV390XAftGVJSKS+xo1ChPdjz6a+40F0w2LyuR1FhOBaWY2Ca1cJyKSUmlpfjQWTPcK7uOS315hZjOAXYFHI6tKRCRPVDUWnDQJ+vePu5r62+5lVd19prtPdvcNURQkIpJPWrQIITF5clgcKVfVdw1uERFJUyIB776b240FFRYiIhGr3lgwVyksREQitvvu0KuXwkJERFIoLYV582Dp0rgrqR+FhYhIBiQS4d9cbSyosBARyYC994YDDsjdoSiFhYhIhiQS8PTTudlYMNKwMLP+ZrbYzJaYWa3LsprZiWa2wMzmm9k91bZfk9y20Mz+YmYWZa0iIlGraiw4ZUrclWy/yMLCzBoDY4ABQBdgmJl1qbFPZ8K6GD3dvStwXnL74UBP4DvAAYQmhkdGVauISCZ07567jQWjPLLoASxx96XJq73HA4ka+5wJjEku04q7r0hud6AF0AxoDjQFPoiwVhGRyFVvLLh+fdzVbJ8ow2IvYFm125XJbdXtC+xrZs+a2Swz6w/g7s8DM4D3kl/l7r4wwlpFRDIikYDPPsu9xoJRhkVtcww11+1uAnQGegPDgDvMrLWZfQvYHygiBExfMztiqxcwG25mFWZWsXKlltcQkezXty+0bJl7Q1FRhkUl0L7a7SK2bmteCUxy943u/iZh2dbOwHHALHdf6+5rgUeA/6n5Au4+1t1L3L2kXbt2kbwJEZGG1Lx5bjYWjDIsZgOdzazYzJoBQ4Gal6NMBPoAmFlbwrDUUuAd4Egza2JmTQmT2xqGEpG8kEjAe+9BRUXclaQvsrBw903ACKCc8If+Pnefb2ajzKw0uVs5sMrMFhDmKC5y91XABOAN4FXgZeBld38oqlpFRDJp4MDcayxo7jWnEXJTSUmJV+RSTItIQevTBz78EF59Nd46zGyOu5ek2k9XcIuIxCCRyK3GggoLEZEYVDUWzJWhKIWFiEgMiovhwAMVFiIikkJVY8FVq+KuJDWFhYhITEpLw7UWudBYUGEhIhKT7t3hm9/MjQWRFBYiIjHJpcaCCgsRkRhVNRacPj3uSrZNYSEiEqM+faBVq+w/K0phISISo1xpLKiwEBGJWWkpvP8+zJ4ddyV1U1iIiMSsqrFgNp8VpbAQEYnZ7rvDEUdk97yFwkJEJAskEjB/PrzxRtyV1E5hISKSBbK9saDCQkQkC3TqBN/5jsJCRERSKC2FZ54JiyJlG4WFiEiWSCTCtRZTp8ZdydYUFiIiWaJ7d9hrr+wcilJYiIhkCbMwFFVenn2NBRUWIiJZpKqx4BNPxF3JVyksRESySO/e2dlYUGEhIpJFqhoLPvRQdjUWVFiIiGSZRCL7GgsqLEREskxVY8FsGoqKNCzMrL+ZLTazJWZ2SR37nGhmC8xsvpndU217BzN7zMwWJu/vFGWtIiLZYrfd4MgjCyQszKwxMAYYAHQBhplZlxr7dAYuBXq6e1fgvGp33wWMdvf9gR7AiqhqFRHJNokELFgAS5bEXUkQ5ZFFD2CJuy919w3AeCBRY58zgTHuvhrA3VcAJEOlibtPS25f6+6fR1iriEhWKS0N/2bL0UWUYbEXsKza7crktur2BfY1s2fNbJaZ9a+2/WMze8DM5prZ6OSRiohIQci2xoJRhoXVss1r3G4CdAZ6A8OAO8ysdXJ7L+BC4LvA3sDpW72A2XAzqzCzipUrVzZc5SIiWSCRgGefzY7GglGGRSXQvtrtIuDdWvaZ5O4b3f1NYDEhPCqBuckhrE3AROCQmi/g7mPdvcTdS9q1axfJmxARiUtVY8EpU+KuJNqwmA10NrNiM2sGDAVqrjA7EegDYGZtCcNPS5OP3c3MqhKgL7AgwlpFRLLOIYdAUVF2DEVFFhbJI4IRQDmwELjP3eeb2SgzS07dUA6sMrMFwAzgIndf5e6bCUNQT5jZq4QhrdujqlVEJBtVbyy4bl3MtbjXnEbITSUlJV5RURF3GSIiDaq8/Mv2H4MHN/zzm9kcdy9JtZ+u4BYRyWLZ0lhQYSEiksWaN4cBA+JvLKiwEBHJcokEfPABvPBCfDUoLEREstzAgdCkSbxDUQoLEZEs17p1/I0FFRYiIjmgtBQWLoTXX4/n9RUWIiI5IJFswxrX0YXCQkQkB3TsCAcdpLAQEZEUEgl47jmIo2+qwkJEJEfE2VhQYSEikiMOPji+xoIKCxGRHFHVWPCxxzLfWFBhISKSQxIJ+PxzePzxzL6uwkJEJIf07g277JL5oSiFhYhIDmnWLJ7GggoLEZEck0jAihXwn/9k7jUVFiIiOWbAgMw3FlRYiIjkmDgaCyosRERyUCIBixbBa69l5vUUFiIiOai0NPybqaMLhYWISA7q2BG6dYPJkzPzegoLEZEclcnGggoLEZEcVVoarrV4+OHoX0thISKSow4+GNq3z8y8RZPoX0JERKJgBj/9aegVFbVIjyzMrL+ZLTazJWZ2SR37nGhmC8xsvpndU+O+Xcw/CtbWAAAF00lEQVRsuZndHGWdIiK5auRI+MMfon+dyI4szKwxMAY4GqgEZpvZZHdfUG2fzsClQE93X21mX6/xNFcCM6OqUURE0hPlkUUPYIm7L3X3DcB4IFFjnzOBMe6+GsDdV1TdYWbdgT2AxyKsUURE0hBlWOwFLKt2uzK5rbp9gX3N7Fkzm2Vm/QHMrBFwLXBRhPWJiEiaopzgtlq2eS2v3xnoDRQBT5vZAcApwFR3X2ZW29MkX8BsODAcoEOHDg1QsoiI1CbKsKgE2le7XQS8W8s+s9x9I/CmmS0mhMdhQC8z+znQEmhmZmvd/SuT5O4+FhgLUFJSUjOIRESkgUQ5DDUb6GxmxWbWDBgK1LwwfSLQB8DM2hKGpZa6+/+5ewd37wRcCNxVMyhERCRzIgsLd98EjADKgYXAfe4+38xGmVmyBRblwCozWwDMAC5y91VR1SQiIvVj7vkxelNSUuIVFRVxlyEiklPMbI67l6TcL1/CwsxWAm/HXUc9tAU+jLuIDNN7Lgx6z7mho7u3S7VT3oRFrjKzinRSPZ/oPRcGvef8okaCIiKSksJCRERSUljEb2zcBcRA77kw6D3nEc1ZiIhISjqyEBGRlBQWWcTMLjQzT17NntfMbLSZLTKzV8zsQTNrHXdNUUhnTZd8YmbtzWyGmS1MrlHzi7hryhQza2xmc80sA4ucZp7CIkuYWXvC2h/vxF1LhkwDDnD37wCvEdY1ySvV1nQZAHQBhplZl3iritwm4AJ33x/4H+DsAnjPVX5B6FaRlxQW2eN64GK27sybl9z9sWRLGIBZhEaT+SadNV3yiru/5+4vJr//lPDHs+bSBHnHzIqAQcAdcdcSFYVFFkj2ylru7i/HXUtMfgw8EncREUhnTZe8ZWadgIOB/8RbSUbcQPiwtyXuQqISZYtyqcbMHge+UctdI4FfA8dktqLobes9u/uk5D4jCUMXd2eytgxJZ02XvGRmLYEy4Dx3XxN3PVEys8HACnefY2a9464nKgqLDHH3o2rbbmYHAsXAy8mFnoqAF82sh7u/n8ESG1xd77mKmZ0GDAb6eX6ew53Omi55x8yaEoLibnd/IO56MqAnUGpmA4EWwC5m9i93PyXmuhqUrrPIMmb2FlDi7rnWjGy7JJfQvQ440t1Xxl1PFMysCWHyvh+wnLDGy8nuPj/WwiJk4RPPP4GP3P28uOvJtOSRxYXuPjjuWhqa5iwkLjcDrYBpZvaSmd0ad0ENra41XeKtKnI9gVOBvsn/ry8lP3FLjtORhYiIpKQjCxERSUlhISIiKSksREQkJYWFiIikpLAQEZGUFBYi28HM1u7g4yeY2d7J71ua2W1m9kayQ+tTZnaomTVLfq+LZiVrKCxEMsTMugKN3X1pctMdwEdAZ3fvCpwOtE02HXwCOCmWQkVqobAQqQcLRpvZPDN71cxOSm5vZGa3JI8UHjazqWZ2QvJh/wdU9cTaBzgU+I27bwFIdqedktx3YnJ/kaygw1yR+jke6AYcBLQFZpvZU4QrmDsBBwJfJ1y5fWfyMT2Be5PfdwVecvfNdTz/POC7kVQuUg86shCpn+8B97r7Znf/AJhJ+OP+PeB+d9+SbAQ5o9pj9gTS6oOVDJENZtaqgesWqReFhUj91NZ+fFvbAdYRupICzAcOMrNt/Q42B9bXozaRBqewEKmfp4CTkusutwOOAF4AngGGJOcu9gB6V3vMQuBbAO7+BlAB/C7ZqRUz62xmieT3bYCV7r4xU29IZFsUFiL18yDwCvAyMB24ODnsVEZYx2IecBthlbhPko+ZwlfD4wzC4lBLzOxV4Ha+XO+iDzA12rcgkj51nRVpYGbW0t3XJo8OXgB6uvv7ZrYTYQ6j5zYmtque4wHgUndfnIGSRVLS2VAiDe9hM2sNNAOurFrx0N3XmdnlhHW436nrwWbWDJiooJBsoiMLERFJSXMWIiKSksJCRERSUliIiEhKCgsREUlJYSEiIikpLEREJKX/B9CBCi2I6ZMnAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fd5b456a438>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#需要调优的参数\n",
    "C_s = np.logspace(-5, 5, 10)# logspace(a,b,N)把10的a次方到10的b次方区间分成N份  \n",
    "#penalty_s = ['l1','l2']\n",
    "\n",
    "accuracy_s = []\n",
    "for i, oneC in enumerate(C_s):\n",
    "#    for j, penalty in enumerate(penalty_s):\n",
    "    tmp = fit_grid_point_Linear(oneC, X_train, y_train, X_test, y_test)\n",
    "    accuracy_s.append(tmp)\n",
    "\n",
    "x_axis = np.log10(C_s)\n",
    "#for j, penalty in enumerate(penalty_s):\n",
    "pyplot.plot(x_axis, np.array(accuracy_s), 'b-')\n",
    "    \n",
    "pyplot.legend() \n",
    "pyplot.xlabel( 'log(C)' )                                                                                                      \n",
    "pyplot.ylabel( 'accuracy' )\n",
    "pyplot.savefig('SVM_Otto.png' )\n",
    "\n",
    "pyplot.show()\n",
    "  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将C的范围进行了设置，但最大的准确率为0.759左右，对C的可选范围又进行了几次调整，仍然有些令人不太满意"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "RBF核SVM正则参数调优\n",
    "\n",
    "RBF核是SVM最常用的核函数。 RBF核SVM 的需要调整正则超参数包括C（正则系数，一般在log域（取log后的值）均匀设置\n",
    "候选参数）和核函数的宽度gamma C越小，决策边界越平滑； gamma越小，决策边界越平滑。\n",
    "\n",
    "\n",
    "这里我们用校验集（X_val、y_val）来估计模型性能\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.svm import SVC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fit_grid_point_RBF(C, gamma, X_train, y_train, X_test, y_test):\n",
    "    \n",
    "    # 在训练集是那个利用SVC训练\n",
    "    SVC3 =  SVC( C = C, kernel='rbf', gamma = gamma)\n",
    "    SVC3 = SVC3.fit(X_train, y_train)\n",
    "    \n",
    "    # 在校验集上返回accuracy\n",
    "    accuracy = SVC3.score(X_test, y_test) \n",
    "    \n",
    "    print(\"accuracy: {}\".format(accuracy))\n",
    "    return accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.7142857142857143\n",
      "accuracy: 0.7272727272727273\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.7467532467532467\n",
      "accuracy: 0.7207792207792207\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6493506493506493\n",
      "accuracy: 0.7402597402597403\n",
      "accuracy: 0.6818181818181818\n",
      "accuracy: 0.6428571428571429\n"
     ]
    }
   ],
   "source": [
    "#需要调优的参数\n",
    "C_s = np.logspace(-1, 2, 4)# logspace(a,b,N)把10的a次方到10的b次方区间分成N份 \n",
    "gamma_s = np.logspace(-5, 2, 4)  \n",
    "\n",
    "accuracy_s = []\n",
    "for i, oneC in enumerate(C_s):\n",
    "    for j, gamma in enumerate(gamma_s):\n",
    "        tmp = fit_grid_point_RBF(oneC, gamma, X_train, y_train, X_test, y_test)\n",
    "        accuracy_s.append(tmp)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从上述部分运行结果来看，可能是gamma参数设置不合适（gamma越大，对应RBF核的sigma越小，决策边界更复杂，可能发生了过拟合） 所以后边调小了gamma值，但是并没有发现有什么很大的区别，没有比线性模型更有优势"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs3Xl8TNf7wPHPySYLIvYQe21ZZBE7behmKVpVqnyr2tKF7lS1WqqUtrT02/3XTaulqq2dUrV97RFLrZVYI0EEkT2TzPP74yYjkQlBJjPhvF+veTH33nPvmUlmnpx7znmOEhE0TdM07Uqc7F0BTdM0zfHpYKFpmqZdlQ4WmqZp2lXpYKFpmqZdlQ4WmqZp2lXpYKFpmqZdlQ4WmqZp2lXpYKFpmqZdlQ4WmqZp2lW52LsCJaVq1apSv359e1dD0zStTNm+fftZEal2teNummBRv359IiMj7V0NTdO0MkUpdaw4x+nbUJqmadpV6WChaZqmXZUOFpqmadpV6WChaZqmXZUOFpqmadpV6WChaZqmXZUOFpqmadpV3TTzLDRNs7G0c7BrNrh6gHddqFQHvOuAm6e9a6aVAh0sNE27MhHY+TOsfBPSEgvv96x6KXBUqnvp37xtHpVKv85aidPBQtO0op3eC0tegeObwK81DPodvKrBheOQdCLfvycg4QAcWgHZGQXPUa5i4QBSqc6l1olXNVDKPq9PKzYdLDRNKywzGdZMgc2fg7s39PoEQgaCU243p3dtoF3hciKQehaSjhsBJC+g5P3/2EbITCpYxsUDvP2sBJLcYFLBF5ycbf6StSvTwULTtEtEYN98WP46JMdB2GC4azx4Vi5eeaWgfDXjUbul9WMykozgYa11Er8b0s4WPN7JBSrWutQSsdzqyg0u3n7gUu5GXrVWDDpYaJpmSIyBpSMh5m+oGQT9foA6rUr+Ou7eUNMbagZa35+VBkmx1lsnR9ZBcjyIOV8BBeVrFHGbK3ebm1fJv45bjA4WmnarM6XD/z4yHi7u0PU9aPUkONvp68HNE6o1MR7W5Jjg4snLWicnjOByMgr2LQSzqWAZj8oFO+Evb514+Oh+k6vQwULTbmWHVhqtifNHIeghuGciVKhp71pdmbMr+NQ3HtaYzZByqmCrJC+gnD1ktJxMaQXLuJUvHEDyt068ql3qr7lF6WChabeipFhY/hrsXwRVGsOjC6HhHfauVclwcjL6OCrWAtoU3i9izBnJu811eevkxBbIuFCwjHO5yzrhL2udVKhlv5ZYKbm5X52maQXlmGDTp7D2PeNL8863oN2IW6uDWCnwqmI8aoVaPybjYr4AcllA+fdPSD1z2Tmdczvhixgi7O0Hru62f202ZNNgoZTqCswAnIGvRWTKZfs/AjrnPvUEqotIpXz7KwL7gT9EZIQt66ppN72jG2DJy8Z8iKbdoesU8Kln71o5JveK4B4ANQKs7zdlFOyEzx9Mjm2Af05e1gmP0Qlf4DbXZa2TchVs/7pugM2ChVLKGfgUuBuIBbYppRaKyL68Y0TkpXzHPwdcHubfAdbaqo6adktIOQMr3oTdc4y/ch+eDc2627tWZZurO1S9zXhYk5NtDD2+cLxw6yR+NxxYAjlZBcu4Vyo8iit/68Szsl074W3ZsmgNRIvIYQCl1BygN7CviOMHAOPyniilWgI1gOVAuA3rqWk3J3MORH4Lq94xOnQ7jYROr+hcTqXB2eVSy8Eas9m4lZXXT5I/oJw7DEfWQlZKwTKuXpcFkHxzTnzq2Xxggi2DRW3gRL7nsVjtbQKlVD2gAfB37nMnYBrwH+BOG9ZR025OJ7fD4pchfic0uAO6Ty16KKpW+pycjC/3CjWtz2URgfTzhWfA57VOTm6H9HOXjvcNhqfW2bTKtgwW1tpLUsSxDwPzRCQn9/mzwFIROaGu0OxSSg0DhgHUrVtEBNe0W0n6eaMlEfktlK8OD34DgQ/qOQRljVLGbSfPykYgsCYz5VI/ibL9sF5bBotYoE6+535AXBHHPgwMz/e8HdBJKfUsUB5wU0qliMhr+QuJyFfAVwDh4eFFBSJNu/mJGOnDV7xp/MXZ5mnoPMaYLa3dnMqVh+rNjUcpsGWw2AY0Vko1AE5iBIRHLj9IKdUU8AE25W0TkYH59j8GhF8eKDRNy3V6X25m2I1GZtgef4BvC3vXSrvJ2CxYiEi2UmoE8CfG0NlvRWSvUmoCECkiC3MPHQDMERHdMtC0a5GZAmunwKbPjKGevf4LIYNu+ZnGmm2om+U7Ojw8XCIjI+1dDU2zPRHYvxCWjzFyJIU9CneONyaZado1UkptF5GrjjjVM7g1rSxJjIFlr0L0X1AjCB76Huq0tnettFuADhaaVhaYMi5lhnV2M2Zftxp60+cj0hyH/k3TNEd36K/czLBHjGGw90yCir72rpV2i9HBQtMcVVKs0S+xf2FuZtgF0DDC3rXSblE6WGiao8kxGWtfr5kCkgNd3oT2z91amWE1h6ODhaY5kqMbjDkTCfuhSTfoNqXoRX40rRTpYKFpjiAlAVa+aczC1plhNQekg4Wm2ZM5B7Z/B6smQFaakRW200idGVZzODpYaJq9nIwyFiOK2wENbofu03RmWM1h6WChaaUt/Tz8PRG2faMzw2plhg4WmlZaRGDXHKNvIi0R2jwFnV/XmWG1MkEHC00rDWf2G6Ocjm2A2uEw6Lei1ynQNAekg4Wm2VJmCqx9DzZ/BuUqQM8ZEPqozgyrlTk6WGiaLYjA/kWw/DUjM2zof+Cut3VmWK3M0sFC00raucOw9FWIXgk1AqHvd1DX6vLzmlZm6GChaSXFlAEbpsP6D8HZFe6dDK2H6cyw2k1B/xZrWkmI/guW6Myw2s1LBwtNuxFJJ+HPMbBvAVS5Df4zHxp1tnetNK3E6WChOaTDSYfZfno7NT1rUqt8LXy9fPF0daAUGDkm2PIFrJ6cmxl2LLR/XmeG1W5aOlhoDicjO4MRq0ZwIvlEge0+5XzwLe9LLa9aln9rla9lCSYV3SqiSmMW9LGNxpyJM/ug8b3Q/X2dGVa76elgoTmcr3Z/xYnkE3wY8SFVPaoSlxJHfGo8J1NOEp8ST0xSDP87+T8ycjIKlPNy9cLXy5fa5Wvj6+VrBJF8QaWKe5UbCyYpCbDyLdj1M3jXgYd/hqbddZoO7Zagg4XmUP49/y/f7fmOXo16cXe9uwEIrR5a6DgR4XzmeeJTcoNIajxxKXHEpcYRnxJP1JkokrOSC5Qp51wOXy/fS4Ek999a5WtRy6sW1Tyr4eJk5SNhzoHt38Oqt43MsB1fgttHgZuXLd4CTXNIOlhoDiPHnMPbG9+mglsFRoaPvOKxSikqu1emsntlAqoGWD0mOSvZ0iq5vHWy+sRqzmWcK3C8s3KmhmcNfMvna51kZ+O781dqnT6Ar19b3Hp8CNWalthr1rSyQgcLzWHM/Xcuu8/u5t2O7+Lj7nPD56vgVoGmlZvStLL1L/eM7AziU+ON1kmqEUTyWiZb4zdzJu0MZgA3oE4t4DjVVj9jvd8k91+H6oTXtBKkg4XmEE6nnmZG1Aza+bbjvob3lco13V3caeDdgAbeDS5tFIHdc2HFG5jSEjkdNpD44AeJy7poCSRxqXHsTdzLX8f/ItucXeCc3uW8jUBy2S2uvMDiXc67dDrhNa2E6WChOYTJWyeTbc7mzbZv2u/L9MyB3Myw/4PaLXEdOA+/WiH4FXG4WcycTT9b6BZXXGocxy4eY1P8JtKz0wuU8XDxKNgBn+/f2uVrU8WjCk5KJxnUHI8OFprdrTq+ilXHV/Fi2IvUqVin9CuQmQLr3odNn4Jb+WJnhnVSTlT3rE51z+qEEFJov4iQlJl06RbXZf0nuxJ2cTHrYoEyrk6uRid8vltd+YNLDc8a1jvhNc3G9G+dZlcpWSm8u+Vdmvg04dGAR0v34iJwYDEsew0uxkLooNzMsFVL5PRKKSq5V6KSeyUCqljvhE81pRYIInGpccbzlHjWn1zP2fSzBY7PC1B5fSSW2125gcXXyxd3F/cSqb+m5aeDhWZXH+/4mIS0BKZHTMfVybX0LnzuCCx7FQ6tgOoB0PcbqNu29K6fy8vVi8Y+jWns09jq/sycTE6lnjICSb6hwXGpcUSdjuJ02mlyJKdAmSruVawODc5rrZR3K18aL027ydg0WCilugIzAGfgaxGZctn+j4C8RDqeQHURqaSUCgE+ByoCOcAkEfnFlnXVSt+uhF3MOTCHAc0GEFQtqHQuasqAjR/D+mng5AL3vgutn3LYzLDlnMtRr2I96lWsZ3V/tjmbM2lnrA4RPnj+IGtOrCHLnFWgTAW3CoVmv+cPKD7lfHQnvFaIEhHbnFgpZ+Bf4G4gFtgGDBCRfUUc/xwQKiKPK6WaACIih5RStYDtQHMRuVDU9cLDwyUyMrLEX4dmGyaziX6L+pGclcyC+xfg5VoKE9yiV8HSkcZ6EwEPGIGiYi3bX9eOzGLmXMa5Ai2TywNLqim1QBkPF48C/Sb5hwb7VfCjqkfJ3KbTHINSaruIhF/tOFv+OdUaiBaRw7kVmgP0BqwGC2AAMA5ARP7N2ygicUqpM0A1oMhgoZUtM/fOJPpCNDM6z7B9oLgYB8vHwL75ULkR/OcPaNTFttd0EE7KiaoeVanqUZUW1VoU2i8iXMy6WOgWV97M+L1n93Ihs+DHrm+Tvrza6lU8XDxK62VoDsCWwaI2kD8TXCxgdbkwpVQ9oAHwt5V9rTGmRcXYoI6aHRy/eJwvdn3BXXXvoktdG35p55hgy5ewZjKYs6HzWOigM8Pmp5TCu5w33uW8aV6ludVj0kxplpbIxriNzNo/i51ndvL+7e8X2dei3XxsOaDb2k3Pou55PQzMEynYU6eU8gV+BIaIiLnQBZQappSKVEpFJiQk3HCFNdsTESZsnoCrkyuvtX7Ndhc6tgm+vANWvAH12sOzm+GOUTpQXAdPV08aVWpEJ79OjG49mi/v+pJzGecYsGQAcw/OxVa3sjXHYstgEQvkHzTvB8QVcezDwOz8G5RSFYElwFgR2WytkIh8JSLhIhJerVq1EqiyZmuLDy9mS/wWXgx7kRpeNUr+AqlnYf5w+K4rZCRB/5/gkblQucHVy2rF0r52e37r9Rsta7Tknc3v8MraV0jKTLJ3tTQbs2Ww2AY0Vko1UEq5YQSEhZcfpJRqCvgAm/JtcwP+AH4QkV9tWEetFJ3POM/7294nuFowDzV9qGRPbjZD5Lfw35awe46RGXbEVmh+n04hbgNVPary+V2f83LLl1l9fDUPLXqIHWd22Ltamg3ZLFiISDYwAvgT2A/MFZG9SqkJSqle+Q4dAMyRgm3ZfsDtwGNKqZ25j8JTZLUyZWrkVFKyUhjXblzJprSI2wnf3AWLX4KaQfD0BrhrvE4hbmNOyokhgUP4odsPOCtnhiwfwle7vyLHnHP1wlqZY7Ohs6VND511bJvjNzN0xVCGBg3l+bDnS+ak6Rdg9STY9jV4VoV7J0HQQ7olYQfJWcm8s/kdlh1ZRuuarZncaTLVPavbu1paMRR36KwOFprNZWRn0GdhHxSK33r9duPpKETgn1/hzzcg7Sy0ehI6vwEelUqmwtp1ERHmR89n8tbJuDu7M7HjRG73u93e1dKuorjBQqe31Gzuy91fciL5BG+1e+vGA8WZAzCzJ/w+FCrVgaGrofsHOlA4AKUUDzR+gDn3zaGaZzWGrxrOe1vfIysn6+qFNYeng4VmU/+e/5fv93xP70a9aeNrdZpN8WSlwspx8EUHOPUP3PcRPPEX1NJdWY6moXdDfu7xM480e4RZ+2cxaOkgjiYdtXe1tBukg4VmM9eyTGqRRGD/Yvi0DWyYDi0ehue2Q/jjV00hrtlPOedyjGkzhhmdZxCXGke/xf1YGFNoMKRWhuhPm2Yzecukjmo1ikru13Gb6NwR+Lk//DIQylWAIcvh/k9LLIW4Zntd6nZhXs95+Ffx543/vcGY9WMK5aLSygYdLDSbuOFlUnf+DJ+1hWMb4J5J8NQ6qNeu5Cuq2VxNr5p8c883PBvyLEuPLKXfon7sTdxr72pp10gHC80mJm+dTI45hzfbXccyqSejYOHz4NcKhm+F9iPAuRTXutBKnLOTM88EP8M393xDZk4mg5YOYubemZgLZ/HRHJQOFlqJW3XMWCb16eCnqVPhGpdJTb8Avz4G5WtAvx/Au7ZN6qjZR3jNcOb1nEen2p2YGjmVEatGkJieaO9qacWgg4VWom5omVQRWPgcXDwJD30HnpVtU0nNriq5V2JG5xm83uZ1tsRvoe+ivmyOt5r+TXMgOlhoJWpG1AwS0hMY3278tS+Tuu1r2L8Q7nwL6rS2TQU1h6CUYkCzAfzc42cquFVg2IphzIiagclssnfVtCLoYKGVmF0Ju/jl4C880vyRa18mNW4n/Pk6NL4H2j1nmwpqDqdp5abM6TGHBxo/wNf/fM1jyx/jZMpJe1dLs0IHC61EmMwmxm8cT3XP6jwXeo1f9hkXjX4Kz6pw/xd6/sQtxtPVk7fbv80Ht3/A4QuHeWjhQ/x59E97V0u7zE2dG8pkMhEbG0tGRoadanXrSM5KJjkrmcrula89pUfqWTClQ/nqenGiW1y2OZsLmRfIysnC09UTbzfvq46mc3d3x8/PD1dXPWLuejjCGtx2FxsbS4UKFahfv/61D9/Uii0zJ5OYCzH4uvle++in1LOQlAEVGkOFmrapoFammMVMQloCZ9PP4ubshl8FvyL/ABEREhMTiY2NpUEDvcCVLd3U7f2MjAyqVKmiA4UNiQjxKfEoFDU9r/HL3pQGSbHG7OzyNlg1TyuTnJQTNbxqUK9iPXIkh8NJhzmXcc7q8q1KKapUqaLvHpSCmzpYADpQ2FhSZhKpplRqeNbA9Vomzplz4NxRcHKBSvX0GhRaIeXdytOwUkO8XL2IT4knNiWWbHN2oeP0Z7x03PTBwlG0adOGkJAQ6tatS7Vq1QgJCSEkJISjR49e03l+//13Dhw4cM3X79ixIzt37rzmcnmmTp3Kzz//XGBbtjmbU2mn8HD1wMfdp/gnE4GkE5CTCT71Smx29kMPPcThw4et7ouOjsbDw8Pyvg8fPtzqcYmJidx55500btyYe++9l6Qkvba0Pbk6uVK3Ql1qeNUgOTOZwxcOk2ZKs3e1bkk6WJSSLVu2sHPnTiZMmED//v3ZuXMnO3fupH79+td0nusNFjfCZDLx448/0r9//wLbT6eexixmannVura/7tISIf08VPA1bkGVkKeffpoPPvigyP1Nmza1vO+ffvqp1WMmTZpEt27dOHToEJ06deL9998vsfpp10cpRVWPqjTwbgAKjiQdISEtweptKc12dLBwAMuWLaNdu3aEhYXRv39/UlONrJyjRo3C39+fFi1aMHr0aNavX8/SpUt56aWXrqtVkmfWrFkEBQURGBjI66+/btn+5Zdf0qRJEyIiInjyySd58cUXAVi5ciWtWrXC2dkZgM2bNxMYFEiPO3vwyTuf0Da8LQAxMTF06tSJ0NBQWrZsyZYtWwD466+/6Ny5M3379qVx49sY+/oYfvjjL1p16UmLFi0sr2PQoEEMHz6czp0706hRI9atW8fgwYNp1qwZTzzxhKWew4YNIzw8nICAACZMmGDZHhERwfLly8nJuf41oBcsWMDgwYMBGDx4MPPnz7/uc2kly8PVg0bejfAu582ZtDMcu3gMU46exFdaburRUPm9vWgv++Iulug5/WtVZFzPgBs6x5kzZ5gyZQqrVq3C09OTSZMmMWPGDJ544gmWLl3K3r17UUpx4cIFKlWqRPfu3enbty/333//dV0vNjaWsWPHEhkZibe3N3fddReLFy8mODiYKVOmEBUVhZeXFxEREbRubcyi3rBhAy1btrScY8iQIbzz33cIbhnMFxO/sGz39fVl5cqVuLu7c+DAAQYPHmwJGLt27WL/3j14m05TP/xunn12ONu2bWPatGl88sknTJ06FYCkpCRWr17Nb7/9Rs+ePdm0aRPNmjUjLCyMPXv2EBgYyJQpU6hcuTLZ2dmWIOTv74+zszP169dnz549BAcHF3rt0dHRhIaG4u3tzbvvvkv79u0LHZOYmEi1atUAqF27NvHx8df1Pmu24ezkTO3ytfFy9eJU6ilikmKoXV7nDysNumVhZxs3bmTfvn20b9+ekJAQfvrpJ44ePUrlypVxcnJi6NCh/PHHH3h5eZXI9bZs2UKXLl2oWrUqrq6uPPLII6xbt86y3cfHBzc3N/r27WspEx8fb/kCPXv2LOmZ6TQPaU4tr1oMHDjQclxmZiZPPPEEgYGBPPzww+zbt8+yr02bNtRwN+HuAg0bNeLebt0BCAoKKtBC6tmzp2V7rVq18Pf3x8nJCX9/f8txs2fPJiwsjLCwMPbv31/gOtWrVycuLq7Q6/bz8+P48ePs2LGD999/n/79+5OSknLV90t3njoepRQ+7j409G6Iq5Mrxy8eJykzSS/famO3TMviRlsAtiIidO3alR9//LHQvsjISFauXMmcOXP4/PPPWbFiRZHnycrKsrQE+vTpw1tvvVXk9a5lO4CHh4dlaGK6KR2zmKlUrhJebgUD2LRp06hTpw6zZs3CZDJRvnx5y75yLgrSz0H5mjg5u1KunDH5zsnJiezsSyNc8m/P+3/+4w4dOsSMGTPYunUrlSpVYtCgQQWGTWZkZODh4cG8efOYOHEiAN9//z0hISG4uxtj9Vu3bk29evWIjo4mJKTgsqxVqlQhISGBatWqcfLkSWrW1HM/HFU5l3I08G7A6bTTnDKdYtDSQbx/+/vU965v76rdlHTLws7at2/P2rVrLaN4UlNTOXToEMnJyVy8eJH77ruPjz76iB07dgBQoUIFkpOTC53Hzc3N0nlbVKAAaNu2LatXryYxMZHs7GzmzJnDHXfcQZs2bVi9ejUXLlzAZDLx+++/W8o0b96c6OhoRASThwkXFxdO7D8BwJw5cyzHJSUl4evri1KKmTNnXgpA2ZnGDG238jc88e7ixYtUqFCBihUrEh8fz59/FkwLcejQIQICAujbt6/l/QgJCSEhIcHSlxEdHc3hw4etTuLq1asXM2fOBGDmzJn07t37huqr2ZaTcsLXy5fK7pX18q02VqxgoZT6TSnVQymlg0sJq1GjBt988w39+/cnODiY9u3b8++//5KUlESPHj0IDg6mS5cufPjhhwAMGDCAd99997o7uP38/JgwYQIRERGEhITQtm1bevToQd26dRk1ahStW7fmnnvuISAgAG9vbwC6d+/O2rVrOZdxjvTsdD7/v88Z+sRQ2rdvj5OTk+W4ESNG8PXXX9O2bVuOHTtmtAzMOZB8ClDgU/+G51OEhYXh7+9PYGAgQ4cOpUOHDpZ9cXFxeHt7W26Z5bd69WpatGhBSEgI/fv35//+7/8s9R4yZIhlWPHrr7/OkiVLaNy4MevWrWPUqFE3VF+tdLi7uDOv5zwCqgTo5VttRUSu+gDuAn4CYoApQLPilCvNR8uWLeVy+/btK7RNK1pycrKIiGRlZUm3bt1k4cKFln097ushf0b+KUeTjsrFixct2ydOnCgvv/xy0Sc9f1TkZJRIepLN6p3n/fffl++//97m19EcT95nPTsnWz7b+Zm0mNlCuv3WTfYk7LFzzRwfECnF+I4tVktBRP4SkYFAGHAUWKmU2qiUGqKU0tm7bhJvvvkmoaGhtGjRgqZNm3LffZfWzn7prZc4feo0vl6+LFq0iJCQEAIDA9m0aRNjxoyxfsK0c8ajfA1wr2jz+lepUoVBgwbZ/Dqa48pbvvXbe7/FZDYxaJlevrWkFDvrrFKqCjAI+A8Qh9HS6AgEiUiErSpYXNayzu7fv5/mzZvbqUY3j4uZFzmRfIIaXjWo6lG1eIVMGXD2ILh6QpXbdDoPzaasfdaTMpN4a8Nb/H3ibzrW7sjEDhOp4lHFTjV0XMXNOlvcPovfgfWAJ9BTRHqJyC8i8hxQ/sqltbIsx5xDfGo87i7uVHYv5jKnZjOcPwLKyUjnoQOFZgfe5byZ3nk6b7R5g63xW/XyrTeouB3Wn4iIv4hMFpECs5SKE5G0sutM2hmyzdn4evniVNzxDRdjITvDSBDo7GbbCmraFSileLjZw/zc42cqulXUy7fegOIGi+ZKqUp5T5RSPkqpZ21UJ81BpJnSOJdxjsoelfF09SxmoXNG7qdS6qfQtOJoWrkps3vMpk/jPnr51utU3GAxVEQu5D0RkfPA0KsVUkp1VUodVEpFK6Ves7L/I6XUztzHv0qpC/n2DVZKHcp9DC5mPbUSYhYzcalxuDi5UN2jevEKmTKMbLKuXkaSQE1zIJ6unoxvP14v33qdihssnFS+vAdKKWfgivcXco/5FOgG+AMDlFL++Y8RkZdEJEREQoD/Ar/nlq0MjAPaAK2BcUqpa8iB7XjKWoryxPREMrMz8fXyxdnJ2WqK8gLMZjh/lJKaT3E9rpSifPny5YSFhREUFETLli1Zs2ZNkef56KOPaNq0Kf7+/gUSLe7cuZO2bdsSEBBAUFAQJpNxKyMzM5Mnn3ySpk2b0qxZswLJB2fPno2/vz8BAQE8+uijlu1Hjx7lrrvuwt/fH39/f06cMCY5igivvfYaTZo0oXnz5gWy465atYrg4GACAgLo0qWLZfu5c+fo06cPzZo1o3nz5mzdurXMvhY/Pz+CgoIICQmhTZs2Rf6MbkTXBl35teevNPBuwMi1Ixm/cTzp2ek2udZNpTjja4EPgF+BO4EuwFxg2lXKtAP+zPd8DDDmCsdvBO7O/f8A4Mt8+74EBlzpemVlnsV3330nw4cPv+7yAwcOlD/++OOay3Xo0EF27NhRrGMzsjNk79m9cvzicREx5l20aNFCsrOziy50/njufIoL11y3kvLXX3/J008/bXXf9u3bJS4uTkREdu7cKX5+flaPW7Fihdxzzz2SkZEhIiKnT58WEeM9CAwMlN27d4uISEJCguTk5IiIyOuvvy7jxo0TEZGcnBw5e/asiIjs379fwsLC5Pz58wXOJSLSsWNHWbVqlYgY81vS0tJEROSrr76SIUOGiNlsLlAmMTFRmjdvLidOnCh0rkd8IwdgAAAgAElEQVQeeUS+++47ERHJzMyUCxculNnXUrt2bcs1rsX1fNazcrLko8iPJOj7IOn9R285eO7gNZ/jZkAx51kUN1g4Ac8A84DfgKcA56uU6Qt8ne/5fzA6yq0dWw+IzzsnMBIYm2//m8BIK+WGAZFAZN26dQu9CWUlWCxdulTatm0roaGh0q9fP0lJSRERkZEjR0rz5s0lKChIXn31VVm3bp34+PhI/fr1JTg4WI4cOVLs6+YPFj/++KMEBgZKQECAjBkzxnLMF198IY0bN5Y2HdrIgwMflOeef05ERJYsWSJPPPGE5bhNmzZJUFCQtGvXTkaOHCnBQYEiJ6Mkesf/pGPHjhISEiJhYWGyefNmERFZuXKlREREyIMPPii33XabvPHGGzJz5kwJDw+XoKAgy+sYOHCgPPvssxIRESENGzaUtWvXyqOPPipNmzaVxx9/3HL9oUOHSsuWLcXf31/efvtty/bs7GypX7/+lYOaGF+ClSpVkqysrEL7HnjgAVm9enWh7QsWLJDBgwdbPV+tWrUsX5D5vfTSS5Yv8fx27dold9xxh9VzhYaGWv25zpgxw/Ilnl9iYqI0atTI6rnK2msRKd1gkWfDyQ0S8UuEtPyxpczZP8cS3G4VxQ0WxZ2UZxaRz0Wkr4g8KCJfisjVFg2wdh+iqEkdDwPz8p2zWGVF5CsRCReRcGspHgpY9hp816NkH8sKdcNcs/wpyqOiomjRogUzZszg9OnTlhTlu3fvZsyYMXTq1Inu3bvz0UcfXdfCSXApRfnq1avZsWMHGzZsYPHixZw4cYIpU6awYu0Kvpj7Bcejj1tGP1lLUf7111+zceNGJMcEOVng6oVv42BWrlzJjh07+Omnn3j++ectZXbt2sWnn37KP//8w9dff83Ro0fZtm0bgwcP5pNPPrEcl5ei/P3336dnz56MHj2affv2sX37dvbs2QPAlClTiIyMZNeuXaxcudKSdTZ/ivIrmTt3Lm3atMHVtfB80n///Zc1a9bQpk0bIiIi2L59u2W7iHDPPfcQFhbGtGnTACMLr5ubG2PGjLGsR5KQkGAps3//fjp06EC7du0siSD//fdfKlasyP33309oaCijR4/GbDYmjR05coRZs2YRHh5O9+7diYmJsZRJTEzkjjvuIDw8nFmzZgFw+PBhqlWrxqOPPkpoaCjDhg0jLS2tTL4WMEYvdenShZYtW/LNN99c8edYUtrXas+8nvMIrxnOxC0TeXnNyyRl6hUSL1fceRaNlVLzlFL7lFKH8x5XKRYL1Mn33A9jMp81DwOzr7NsmeZIKcojOkeQUS6Dil4V6f/QpVXxLk9RbslwK2Ye6dGJvH6KzCzTlVOU16iBu7s7DRs25N577wVKL0V5nn/++YexY8fy+eefW92fnZ1NUlISW7ZsYfLkyZbVAbOzs9mwYQOzZ89m/fr1/PLLL6xdu5bs7GyOHj1K586diYqKomXLlrz66quWMocPH2bt2rXMmjWLxx9/nIsXL5Kdnc369euZPn06W7du5cCBA5aswxkZGVSoUIHIyEgee+wxnnzyScu5oqKiWLZsGcuWLWP8+PHExMSQnZ1NZGQkzz//PFFRUbi5uVlWCyxrryXv9zMqKoolS5Ywffp0Nm7cWOTPsiRV8ajCZ3d+xsjwkaw5sYa+i/oSdTqqVK5dVhQ3Rfl3GB3OHwGdgSFY/+s/v21AY6VUA+AkRkB45PKDlFJNAR9gU77NfwLv5uvUvgejz+P6dZtyQ8VtRcRxUpSnZ6dbXSY1f4ryAuUvnjQyyjq7gosb06ZNKDpF+WXpxu2Rovz48eP06dOHWbNmWc04C0YH64MPPghAu3btMJlMnD9/Hj8/PyIiIqhSxZgB3K1bN6KioujUqROenp706tULMDrZ8xamyivj4uJCo0aNaNSoETExMfj5+dGyZUtLy/D+++8nKiqKwYMHU7t2bcv1H3zwQZ566inLufz8/PD09MTT05MOHTqwe/duWrVqRb169QgPD7eUmT59epl8LY0aNaJWrVoA1KxZk969e7N161ari1TZgpNyYnDAYFrWaMmr615lyJ9DeCb4GYYGDcXZyblU6uDIijsaykNEVmGkBzkmIuMxOrqLJCLZwAiML/79wFwR2auUmqCU6pXv0AHAHMn3LSQi54B3MALONmBC7rabjqOkKA8MDWTDug24ZLrgLM5WU5QDVKtWDVdXVyL/txpSzzJn6XrI/SAVmaK8BF1vivLz58/To0cPpk6dStu2bYs8//3338/ff/8NGCkkAHx8fOjWrRs7duwgPT2d7Oxs1q1bZ2n1dOvWjfXr1wPGKB9/f3/LuVavXg0YtxtjYmJo0KABbdu25cyZMyQmJgLw999/FyiTd/3Vq1fTrFkzy/Z169aRk5NDamoqW7dupVmzZvj5+VG9enXLz+fy65el15KSkmJZkColJYWVK1cSGBh41d+JkhZYNZC5982la/2ufLrzU4auHMrp1NOlXg+HU5yODWADRmD5HSMAPAAcLE7Z0nqU5dFQK1askPDwcGnRooW0aNFCFi9eLCdOnJBWrVpJixYtJDAwUH744QcREVm7dq00a9bshjq4f/jhB0sH92uvvSY55hw5eO6gjJ86Xho3biwREREybNgweeutt0REJCYmRiIiIizn2rB2tQQ2u03ahYfI6Fdfldtvv11ERA4cOCCBgYHSpk0beeONN8TLy0tEjA7u3r17W61L/n35R3odOnRIgoODLWXy9pnNZhk0aJA0b95cevToIb1795Yff/xRREROnjwpbdu2tfr6x40bJ15eXhIcHGx55I30eeyxxyz1ycjIkIcfflgCAgIkLCxM1qxZYznH999/L82bN7e8b3kOHz4sHTt2lKCgILnzzjvl+HFjFJnZbJbnn3/eMkhh7ty5ljLLly+XoKAgCQwMlMcff9zS2Z6YmChdu3aVwMBAadeunWXEkojI5MmTLdf/+OOPLdsjIyMlLCxMgoKC5P7777eMhiprr+Xff/+1fAb8/f1l8uTJVn+W1tjis242m2X+ofnSalYr6Ti7o6w5vubqhcogSng0VCuMHFB+GLekfgPaFqdsaT3KSrBwRKdSTsmehD1yKvGUiFhPUd6zZ0+JiYkRMedI8uHtInG7REwZV09RXop0ivJbly0/64cvHJa+C/tK4PeBMnnLZMnMzrTZteyhuMHiqrehcifX9RORFBGJFZEhYoyI0hm5bgIZ2RmcTT9LJfdKTHlnSpEpyt977z2j4/hiHAuXLifk3oEEhrS8coryUqZTlGu20MC7AT91/4mBzQfy0/6fGLh0IEeSjti7WqWuWCnKlVJ/A3dKcQ62E52i/NqJCEeSjpBlzuK2Srfh4nSV8Q7pF4xssl7VwNuvdCqpacVQWp/1NSfW8OaGN8nMyeSNNm/Qq1GvAoNByqISTVEO7AAWKKX+o5Tqk/e4sSpq9pa3TGpNr5pXDxTZmXDhOLh6QMVapVNBTXMwEXUiLMu3jt0wljH/G0NKVoq9q1UqihssKgOJGCOgeuY+7rtiCc2hmXJMnEk7Q3m38ni7eV/5YMnL+wT4NDDWqdC0W1QNrxp8fc/XDA8ZzrIjy+i3uB97z+61d7VsrljzLERkiK0ropWu+NR4BMHXy/fqzeiL8WBKMxIEupS78rGadgtwdnLm6eCnaV2zNaPXj2bQskG8GPYi//H/T/HXfSljihUslFLfYT3dxuMlXiPN5i5mXiQ5K5kaXjVwu9riRBlJkHoGPKuCR5lO/KtpJS6sRhjzes5j3MZxTI2cyqb4TUzqMOmmXL61uCFwMbAk97EKqAjcGjfqSoijpCi/pmVSs7Pg/DFw8WDq/825copyB3ClFOUAEydO5LbbbqNZs2b89ddfVo8ZPHgwwcHBBAUF0a9fP1JTUwH45JNPLKmzO3XqZPkZbNq0ieDgYEJCQggODmbhwoWAMbmydevWhISE4O/vz4QJE/Q1irgGwNKlS2natCm33XabJV1Jfs888wyVKlUqtN0ReJfz5qOIjxjbZizb4rfRd1FfNsVtunrBsqY442svf2AEmb+vp6ytHmVlnoW9U5SfTD4pexL2SFpW4cyiBZhzRM4cFInbKVlpF6+eotwBXClF+a5duyQ0NFQyMzMlOjpabrvtNkta7vySkpIs/3/uuefkgw8+KLT9t99+kx49eoiISGpqqphMJhExJgVWr15dcnJyJCcnx5I9OCsrS1q2bCnbtm3T17ByjaysLGnQoIEcPXpUMjIyJDAwUA4evJQufPPmzTJo0CDx9vaWojjKZ/3guYPS649eEvR9kHwU+ZFk5RTObOxoKMmss1Y0BuqWVMC61S1btox27dpZMn3m/YU2atQo/P39adGiBaNHj2b9+vUsXbqUl1566bpaJemmdM5nnGf1gtW0DmtNYGBggQVxvvzyS5o0aUJERARPDh7Ii2PGg3cdVq5eT6tWrXB2NtJ6bN68mRYtWtC+fXtGjRpFSEgIADExMXTq1InQ0FBatmzJli1bAPjrr7/o3Lkzffv2pXHjxowdO5YffviBVq1a0aJFC8vrGDRoEMOHD6dz5840atSIdevWMXjwYJo1a8YTTzxhqeewYcMIDw8nICCgwF+6ERERLF++nJycwgmRFyxYwIABA3Bzc6NRo0bUrVvXkoU1v4oVjaVgzWYzGRkZlv6cvO1g/LWdt93T0xMXF+Nubnq6sYCOiODk5GRJ/piVlYXJZCp0Ln0N4xqbN2+mefPm1KtXj3LlytGvXz8WLFgAGEkHR48ezZQpjpnb7XJNfJow57459Gnch2/2fMNjyx8jNjnW3tUqEcXts0imYJ/FKWC0TWpkI+9tfY8D56799s2VNKvcjNGtb+xtyJ+i3NPTk0mTJjFjxgyeeOIJS4pypRQXLlygUqVKdO/enb59+1oSvF2LhIwEsuKzmPrOVCIjI/H29uauu+5i8eLFBAcHM2XKFKKiovByziGiy520bhUOnpWtpiifOXMmrVu3ZuTIkZbtvr6+rFy5End3dw4cOMDgwYMtAWPXrl3s378fb29v6tevz7PPPsu2bduYNm0an3zyCVOnTgUupSj/7bff6NmzJ5s2baJZs2aEhYWxZ88eAgMDmTJlCpUrVyY7O9sShPz9/QukKA8ODi7w2k+ePElERITluZ+fHydPnqRVq1aF3qdHH32U5cuXW9LF5/n444+ZMWMGJpPJkicJjMzBQ4cO5dixY/z888+WoJqX3DE6OpoXXnihwHuor3HpGidPnqROnUtJpv38/Ni1axcAM2bM4MEHH6RGjRqFfk6OysPFg/Htx9O2VlsmbJzAQ4seYlz7cXSt39XeVbshxV3PooKIVMz3aCIiv9m6creC0kpRbjKbMGWbOLHvRJEpyrt06YJPBS/c0uLp26sbuBlZY4tMUQ488silRMKZmZkOm6JcrMwnLWoU2A8//EB8fDyNGjXi119/tWx//vnniYmJYeLEibz77ruW7e3bt2fv3r1s2bKFSZMmkZWVBVxK7njixAk2bNhgSeanr1HwGkX9bGJjY5k/fz7PPvus1Z+To+tavyu/9vqVhpUaMmrtqDK/fGtxWxYPYPRRJOU+rwREiMj8K5d0HDfaArAVEdunKM/MycRkNuHl5oU4F52iHBG4cNSYV+HhA8pI/V1kivLLTJs2zWFTlPv5+VnWhgZjEai8dNjWODs7069fP/773/8WWG8ajAD5wgsvFFqcJyAgADc3N/bt22e5NQdGptdOnTrx559/FphlrK9hXKOon01UVBSHDh2iUaNGgJFxuGnTphw8eJCyonb52nzf9Xs+2/kZ3/zzDTvO7OCDOz6giU8Te1ftmhW3z2JcXqAAEJELGOtbaDfI1inKRYT4lHgUiiruVYpMUd6mTRtW/72KCwmnMHnW5PcFiyznsJqiPDe1ypw5cyzHOXKK8l69ejF79myysrKIiYnh2LFjBW6ngHHvPe/nICIsWrTIklb70KFDluMWLVpE06ZNAWM1uLw+kiNHjhAdHU29evU4c+YMSUnGRyYtLY1Vq1bRrFkzfQ0r12jbti379u3j2LFjZGZmMnfuXHr16kWvXr04deoUR48eJTo6mooVK5apQJHH1cmVF8Je4Mu7v+Ri1kUGLB7ALwd+scnnw5aKu/iRtaBS3LLaFdSoUYNvvvmG/v37W5r97777Lh4eHvTp04fMzEzMZjMffvghAAMGDOCpp55i2rRpzJ8//6pLqyZlJpFqSsXVyRVXZ1f8/PyYMGECERERiAg9e/akR48ekHGRUU8NpHXPx6hdpx4BAQF4exszu7t3716gg/nbb79lyJAhVKhQgdtvv91y3IgRI+jbty+zZ8/mrrvuKtAyKClhYWH4+/sTGBhIw4YN6dChg2VfXFwc3t7eWFtiNzg4mPvvv5/mzZvj4uLCZ599hpOT8Wt977338uOPP+Lj48OgQYMswTgkJITPPvsMgOnTp7NmzRpcXV2pUqUK3333HQBr167lgw8+wNXVFWdnZ7788kt8fHzYuXMnjz32GCJCTk4OAwYMoGvXrphMJn2Ny64BRh/H3XffTU5ODsOGDbMEmJtJu1rtmNdzHmM3jGXilolsit/E2+3fxrvcVTIoOIjiJhL8FrgAfIrR0f0c4CMij9m0dtdAJxIsLNucTfSFaNyc3WhQsUHRM7VzsiDhICnpmZSvF4opx0zv3r155plnLH0IvXr1Yvr06TRs2JCUlBTLLaZJkyZx7tw5yzrO9vTBBx9QvXp1Bg8ebO+qaKWsLH3WzWLmx30/Mj1qOlU9qvJep/cIqxFmt/qUdCLB54As4BdgLpAODL/+6mml4VTqKavLpBYgYky8EzNvTp9JaMvwK6coBxYuXEhISAiBgYE6RbmmXaO85VtndZuFq5MrQ/4cwhe7viDHXHjItyMpVsuiLNAti4JSslI4dvEYVT2rUsPzCsMOL8ZDyimoVBc8b74UBdqtoax+1lOyUpi4ZSJLDi+hVc1WTO44mRpepTtMuERbFkqplbkjoPKe+yil/rxSGc1+zGImLjUON2c3qnkUvn9vkZlsBAqPyjpQaJodlHcrz+SOk5nYYSJ7zu6h76K+rDmxxt7Vsqq4t6Gq5o6AAkBEzgPVbVMl7UYlpCVgyjFRq3ytojNg5piMtOMu7nohI02zI6UUvW/rzdz75uLr5ctzfz/HlK1TyMzJtHfVCihusDArpSzpPZRS9bGShVazv/zLpHq5FjGRT8QIFGazkXbcybk0q6hpmhX1veszq/ssBjUfxE/7f2LQ0kEOtXxrcYPFG8D/lFI/KqV+BNYCjtGrqVmICHEpcTg7OV+5nyLlFGSlGC0KV4/Sq6CmaVfk5uzG6Naj+aTLJ5xKPUX/xf2ZHz3fIeZkFDfdx3IgHDiIMSLqFYwRUVoxlUaK8istk5qXopzMZEg+ZczQ9rxKivJ8pk6dWqZSlCcnJ9O9e3eaNm1KQEAAb7zxRpHlipO6XNNK0x117mBez3kEVg3kzQ1v8tr61+y/fGtxUtMCTwL/AOeB1RiBQqcovw62SlGelZ0l+87uk6NJR8VsNhfa36FDB9kRuVUk/h+RU3tFcoqfbjwrK6vMpShPTk6WNWvWiIhIRkaGtGvXTlasWFGoTHFTl2uOzRE/6yUhOydbvtj5hQTPDJZuv3WTfxL+KfFrUMIpyl8AWgHHRKQzEAoklHjkukXdaIpyESneMqnJ8WDOZtayzQQFh1w5RfmTT/Liiy8CsHLlyjKXorx8+fLccccdgJFvKjQ0lNjYwqmii5u6XNPswdnJmaeCn+K7rt+Rbc7mP0v/w/d7vscs5lKvS3FTdmSISIZSCqVUORE5oJQqU/PxT737Lpn7SzZFebnmzaiZ78v2epREivLkrOSrL5OakwVZacSmODF2/IQrpyj38iIiIsKSmLCspyg/f/48S5cu5dVXXy30tlxL6nJNs5fQ6qH82vNXxm8cz7Tt09h8ajMTO0ykqkfVUqtDcVsWsbnzLOYDK5VSC4DCeaC1a3ajKcrzL5Naxb2IuRKZKcZQ2XIV2bL73yunKPfxwc3Njb59+1qKl+UU5SaTif79+/PKK69Qr169Qm+NXEPqck2zJ+9y3nwY8SFvtn2TyFOR9F3Yl41xG0vt+sVqWYjIA7n/Ha+UWg14A8ttVisbuNEWgK3IDaYoP512mmxzNjXL1SQ0NBS4LEV5TrYxTFYpqFADSbA+FM/al2aespiiPK+uecFrxIgRVut8ranLNc2elFL0a9qP0OqhjFo7iqdXPs2QwCGMCB2Bq5OrTa99zcuqishaEVkoIlm2qNCt5kZSlKeZ0jifcZ7KHpXx9vIunKJcBC4cA3O2MfnOyfnKKcpXr+bChQuYTCZ+//13Sx3LYopygDFjxpCRkWG5xWVNcVKXa5qjaezTmNn3zebBJg/y7Z5vGbpiqM37MWyaZlwp1RWYATgDX4tIoYV0lVL9gPEYk/x2icgjudvfB3pgBLSVwAtii28fO7uRFOXTf5hOLb9aVPcoYjJ9yhnIvGjMp8idyV1kinKMDvXWrVtTu3btMp+i/OjRo7z33ns0b96csDAjo+cLL7zAkCFD+OOPP/jnn3946623rpi6XNMcmYeLB+PajaOtb1uSMpOKztZQUoozZOp6HhgBIgZoCLgBuwD/y45pDOzASHcOUD333/bAhtxzOAObMFbmK/NDZ0vKmdQzsidhj1zMvGj9gIxkkZNRIomHRawMpbUmOTlZRIyhst26dZOFCxda9vXs2VNiYmIKHCciMnHiRHn55Zev81WUrPfff1++//57e1dDs4Ob+bNua5Tw0Nnr0RqIFpHDYtyymgP0vuyYocCnYuSaQkTO5G4XwB0jyJQDXIHTNqxrmZKZk0lCegIVy1WkgluFwgfk9VM4u0GlOkZ/RTG8+eabhIaG6hTlmqYVYsvbULWBE/mexwJtLjumCYBSKq8VMV5ElovIptyO9HhAAZ+IyP7LyqKUGgYMA6hbt+7lu29Kkm+Z1JqeNa0dcKmfomoTcCr+j/ijjz4qcl/+9M+PPPJIgVFQjuLxxx+3dxU07aZly5aFtT9nL+9zcMG4FRUBDAC+VkpVUkrdBjQH/DCCThel1O2FTibylYiEi0i4taU0b0YXMi+QakqlhlcNXJ2tjH5Ize2nqFgb3DxLv4Kapt2UbBksYoE6+Z77UXhuRiywQERMInIEI/dUY+ABYLOIpIhICrAMaHs9lZCbqE8825zN6bTTeLp64lPOp/ABWanGYkbu3uBVepN1NM2ebqbPuCOzZbDYBjRWSjVQSrkBDwMLLztmPtAZQClVFeO21GHgOHCHUspFKeUK3AEUug11Ne7u7iQmJt40v0x5y6RaTelhzuuncDVWvdMTy7RbgIiQmJiIu7u7vaty07NZn4WIZCulRgB/YvRHfCsie5VSEzB63xfm7rtHKbUPyAFGiUiiUmoe0AUjeaEAy0Vk0bXWwc/Pj9jYWBISyn4aq8ycTBLTE6ngVoEjZ6xMrEs9C6Z0KF8Dzh0q/Qpqmp24u7vj56cX8LK1m3oN7ptFenY6fRb0wcXJhXm95lHO+bL5C5s/h+Wvwb2Tod2z9qmkpmllUnHX4LbppDytZHyx6wtiU2L59t5vCweKk9thxZvQtAe0fcY+FdQ07aanp6o6uIPnDjJz70weuO0BWtW8LBNq+gX49TGoUBN6f6L7KTRNsxndsnBgOeYcxm8cj3c5b14Jf6XgThFYOAIuxsGQ5de06p2madq10i0LBzbn4Bz2JO7h1Vav4l3Ou+DOrf8H+xfBneOgjl57QdM029LBwkGdSj3Fx1Ef06FWB7o36F5wZ9xOWPEGNL4X2llPva1pmlaSdLBwQCLCpC2TMIuZsW3HFpxTkXHR6KfwqgYPfAE6Q6qmaaVAf9M4oFXHV7HmxBqGhwzHr0K+8eMisOh5uHAc+n6r+yk0TSs1Olg4mOSsZN7d8i7NKjdjkP9lGVQjv4W9f0CXsVD3urKfaJqmXRc9GsrBzIiaQWJGIh93+RiX/Blj43fD8jFw213Q4UX7VVDTtFuSblk4kJ1ndvLLwV94pNkjBFYNvLQjM9nop/CsDA98qfspNE0rdbpl4SBMOSbe3vQ2Nb1q8lzoc5d2iMCiF+H8ERi8WGeT1TTNLnSwcBDf7f2O6AvR/LfLf/F0zbcORdRM2DPP6Keo36HoE2iaptmQvp/hAI4mHeXLXV9yT717iKgTcWnHqT2wbDQ07AwdXymyvKZpmq3pYGFnIsI7m9+hnHM5Xmv92qUdmSlGP4W7N/T5SvdTaJpmV/obyM4WxCxg66mtvNjyRap55i4NKwJLXoFzMfDg11C+un0rqWnaLU8HCztKTE9kauRUQquH0rdJ30s7dv4Eu+fAHaOhQaGlxzVN00qdDhZ29EHkB6SaUhnXbhxOKvdHcWY/LBlpBInbR9m3gpqmabl0sLCTjSc3suTwEp4IfIJGlRoZG7NSYe5gKFcB+nwNTs72raSmaVouPXTWDtKz05mweQL1K9ZnaIuhl3YsHQVn/4VH50OFGvaroKZp2mV0sLCDz3d9zsmUkwWXSd052+iruP1VaBhhz+ppmqYVom9DlbKD5w7yw94f6NO4z6VlUhMOwpKXoV5HiHjtyifQNE2zAx0sSlH+ZVJfbvmysTErzZhP4eppDJPV/RSapjkgfRuqFOUtk/pep/cuLZO6fDSc2QeDfoOKvvatoKZpWhF0y6KUWJZJrd2Bbg26GRt3/wpRP0DHl43U45qmaQ5KB4tSICJM2py7TGqb3GVSz0bD4hehbjvo/Ia9q6hpmnZFOliUgr+O/8Wa2HzLpJrSjX4KZzd48Btw1ncDNU1zbPpbysaSs5KZvGVywWVSl4+B0//AwHngXdu+FdQ0TSsGHSxsLG+Z1P92+a+xTOqe32D7d9DhBWh8t72rp2maViz6NpQN7Tizw7JMakDVAFDMEMgAABA6SURBVEiMgYUvQJ020OVNe1dP0zSt2GwaLJRSXZVSB5VS0Uopq7PNlFL9lFL7lFJ7lVI/59teVym1Qim1P3d/fVvWtaSZcky8vfFtfL18jWVSTRlGP4WTc24/hau9q6hpmlZsNrsNpZRyBj4F7gZigW1KqYUisi/fMY2BMUAHETmvlMq/cMMPwCQRWamUKg+YbVVXW/h2z7fEJMXw6Z2fGsukLhkJp3bDgDlQqY69q6dpmnZNbNmyaA1Ei8hhEckC5gC9LztmKPCpiJwHEJEzAEopf8BFRFbmbk8RkTQb1rVEHU06yle7v+Keevdwu9/tsHc+bPs/aDcCmnazd/U0TdOumS2DRW3gRL7nsbnb8msCNFFKbVBKbVZKdc23/YJS6nel1A6l1Ae5LZUClFLDlFKRSqnIhIQEm7yIayUiTNg84dIyqeeOwMLnoHZLuHOcvaunaZp2XWwZLJSVbXLZcxegMRABDAC+VkpVyt3eCRgJtAIaAo8VOpnIVyISLiLh1apVK7ma34D50fPZdmobL4W/RDW3ijBvCCgFfb8DFzd7V0/TNO262DJYxAL5b877AXFWjlkgIiYROQIcxAgescCO3FtY2cB8IMyGdS0RiemJTNs+jbDqYTzY+EFYOQ7idkDvz8Cnnr2rp2madt1sGSy2AY2VUg2UUm7Aw8DCy46ZD3QGUEpVxbj9dDi3rI9SKq+50AXYh4MrsEzqgSWw5XNo8ww0v8/eVdM0TbshNgsWuS2CEcCfwH5grojsVUpNUEr1yj3sTyBRKbUPWA2MEpFEEcnBuAW1Sin1D8Ytrf+zVV1LwoaTG1hyeAlPBj1JQ3GGBcOhVijcPcHeVdM0TbthSuTyboSyKTw8XCIjI/+/vXsPjqs87zj+/em+xhfJ94tMjMHh4nKpYyBgQglpEwothAIhGTKQaSClCTNpptAhgQlpSCZAk0yhkIaEMoGZBkJI6lAwpTTcpqQGDAEbY+4MIGEbY2zZRrJkSU//OEf2aiNp15JWR5J/nxnNHu2+u3rePdI+es97zvNm8rPbOts48zdnUl1Rzd2n/pza28+A916Bv3kMph6QSUxmZqWQ9HRELC3WzuU+hkGvZVIfuQaan4bP3O5EYWbjhst9DNGL77+4Z5nUlvfg/26Eoy+CwwovKTEzG7s8shiCXsukHnQu3PrnMPsI+OR3sg7NzGxYeWQxBHe+dCdrN6/l8qWXMeWer0J3F5zzM6iuyzo0M7Nh5ZHFIK3fsZ7rn7meZfOWccprT0DTk3D2rTDtwKxDMzMbdh5ZDEJE8N0nvgvAlTNPRL+7AZb+NfzRWRlHZmZWHk4Wg/Dgmw/yaNOjfOXgz9N4/xUw63D41PeyDsvMrGycLPbSto5tXPPkNRzacAjnrb4Pujo8T2Fm456TxV66/ulkmdSrquZS9dZK+It/hukHZR2WmVlZOVnshd+/+3vuevkuzpvzMRY/+TNYcj4ccU7WYZmZlZ2TRYl2L5Oam8klz94PMw+DU67NOiwzsxHhZFGinmVSr/wgmNDRmsxT1EzIOiwzsxHhZFGCN1re4ObVN/Op3HxOfPNpOO2HMOPgrMMyMxsxThZFRARXr7yaOlVx+Usr4ajPw1GfyzosMzM6Ort5a3Mrr2zcXvaf5Su4i+hZJvWb2zqYPnURnHpd1iGZ2T6iraOL5q1tNG1ppXlrG81b2nbfNm1pY+P2nUTAUfPrWf6VZWWNxcliAJvbNvP9Vd9nSdRyVssm+NJyqNkv67DMbJxoaduVlwBaaerZThPC5g86erWvqhBz6uuYV59j2UHTmdeQo7E+x8IZ5f9ccrIYwHVPXUdrxw6uam6i4rTrYeahWYdkZmNERLD5g44kAWxpo3lr6+7E0HPf9vbOXs+prapIEkDDBBbPnUJjQ4559TnmpbezJtdRWaFM+uNk0Y/Hmx9nxRsr+NstLSw89Gw46rysQzKzUaSrO9i4bWfeYaHWPYlgaxvvbG1j567uXs+ZVFfFvPocjQ05jj1gKo0NE3YngnkNOabtV4OUTTIoxsmiD627Wrn6d99iQWc3F1bNhNN+AKN0B5pZebR3drF+a14yKEgKG1p20tnde1nqafvV0NiQ45DZk/jEITPTJDAhGSE05JhcV51Rb4bOyaIPP372X2lu3cCt77dQc/5dUDsx65DMbJi1dnTuTgJ7DhUlcwfNW9t4d3s7kZcLKgSzJifzBR/5UEN6iChvZFCfI1dTmV2HyszJosCL77/I7S/cxlnbd3D0yd+BWYuzDsnM9lJE0NK2a8+Ecf5ZROncwZbWXb2eU10p5qYf+icumrE7CTSmI4PZU+qortx3rzZwssjT1d3Ftx65lCldnXxt9p/AkguyDsnM+hARbNrRvvsU0sKE0Ly1jR0Fk8e56srdCeDIxvq8ZJCMEGZOqqUio8njscDJIs8dq3/K2u1vcl17DVP+8l88T2GWkc6ubjZs2/kHCSD/1NKOzt6Tx1Ny1cyrz7H/tAkcd+A0GhtyvQ4VNUyoHrWTx2OBk0Vq/fZmbnjuR5zQ1s4pn74baidlHZLZuLVzVxfrW3Ymk8W9DhEltxu27aSrYPJ4+sRaGhtyHDZ3Mp88bFavs4jm1eeYNIYnj8cCJwvSZVL/6yLo7uLKwy9Gc47IOiSzMW1He2evawsKJ5E3bW/v1b5CMGdK8qF/zAFT9xweShPB3PocddXjd/J4LHCyAB5cdROPtr7NpbX7M+/4r2UdjtmoFhFsbd21uwxF4ZxB05Y2Wtp6Tx7XVFYwt76OxoYJnHzwzF6jgsaGHLMn11G1D08ejwWKiOKtxoClS5fGqlWr9vp5TRteZfnfn84BG4PoWkg3/u/FrD8R0NHVTXfBIaLKClFTVUFtVSW1VRXUVifbyX0VVFdW4NmC8qk99BBmf+Mbg3qupKcjYmmxdvv8yKKzu50p3VXkNJ2umpqswzEb1YSor0wSQH5SqKyQk8E4t88niwVzF7PgjtVZh2FmNqqV9SChpFMkvSTpVUmX99PmM5JekLRW0s8LHpssqVnSjeWM08zMBla2kYWkSuAm4M+AJuApSfdExAt5bRYBXweWRcQWSTMLXuZq4NFyxWhmZqUp58jiGODViHg9IjqAO4EzCtpcBNwUEVsAIuLdngckfQSYBfx3GWM0M7MSlDNZzAPezvu+Kb0v34eBD0t6XNJKSacASKoAfgBcNtAPkPQlSaskrdq0adMwhm5mZvnKmSz6Ojmi8DzdKmARcBLwOeAWSfXAl4EVEfE2A4iIn0TE0ohYOmPGjGEI2czM+lLOs6GagPl53zcC7/TRZmVE7ALekPQSSfI4DviYpC8DE4EaSTsios9JcjMzK69yjiyeAhZJOkBSDfBZ4J6CNsuBjwNImk5yWOr1iDgvIvaPiAXApcDtThRmZtkpW7KIiE7gEuABYB1wV0SslfRtSaenzR4ANkt6AXgYuCwiNpcrJjMzG5xxU+5D0ibgzSG8xHTgvWEKJ0vjpR/gvoxW46Uv46UfMLS+fCgiik76jptkMVSSVpVSH2W0Gy/9APdltBovfRkv/YCR6YvLPJqZWVFOFmZmVpSTxR4/yTqAYTJe+gHuy2g1XvoyXvoBI9AXz1mYmVlRHlmYmVlR+2yykHROWha9W1K/ZxGUUmY9S5KmSnpQ0ivpbUM/7bokPZt+FV4cmali77GkWkm/SB9/QtKCkY+yNCX05QuSNuXtiwuziLMYSbdKelfS8/08Lkk3pP1cLWnJSMdYihL6cZKklrz98c2RjrFUkuZLeljSuvSz66t9tCnffomIffILOBQ4GHgEWNpPm0rgNWAhUAM8BxyWdewFMV4HXJ5uXw5c20+7HVnHOtj3mKRW2I/T7c8Cv8g67iH05QvAjVnHWkJfTgSWAM/38/ipwP0kNeA+CjyRdcyD7MdJwL1Zx1liX+YAS9LtScDLffx+lW2/7LMji4hYFxEvFWlWSpn1rJ0B3JZu3wZ8OsNYBqOU9zi/j3cDn5A0GlfxHAu/LyWJiMeA9wdocgZJGZ6IiJVAvaQ5IxNd6Urox5gREesj4pl0eztJZYzCSt5l2y/7bLIoUSll1rM2KyLWQ/LLBBQuINWjLi3nvlLSaEoopbzHu9tEUkamBZg2ItHtnVJ/X85KDxHcLWl+H4+PBWPhb6NUx0l6TtL9khZnHUwp0kOxfww8UfBQ2fbLuF6DW9L/ALP7eOiKiPhNKS/Rx30jfvrYQP3Yi5fZPyLekbQQeEjSmoh4bXgiHJJS3uNRsR9KUEqc/wncERHtki4mGTGdXPbIht9Y2SfFPENS7mKHpFNJipsuyjimAUmaCPwK+LuI2Fb4cB9PGZb9Mq6TRUT86RBfopQy62U3UD8kbZQ0JyLWp8PNd/tqFxHvpLevS3qE5L+S0ZAsSi1lPx9oklQFTGF0Hloo2pfoXSjzp8C1IxBXOYyKv42hyv+wjYgVkn4kaXpEjMqaUZKqSRLFv0fEr/toUrb94sNQAyulzHrW7gEuSLcvAP5gxCSpQVJtuj0dWAa8UNguI6W8x/l9PBt4KNLZvFGmaF8Kjh+fTnLceSy6Bzg/Pfvmo0BLz+HQsUTS7J75L0nHkHwmjsrK12mc/wasi4gf9tOsfPsl6xn+rL6AM0mycDuwEXggvX8uySp9+WcXvEzyX/gVWcfdRz+mAb8FXklvp6b3LwVuSbePB9aQnJ2zBvhi1nEX9OEP3mPg28Dp6XYd8EvgVeBJYGHWMQ+hL98D1qb74mHgkKxj7qcfdwDrgV3p38kXgYuBi9PHBdyU9nMN/ZxRmPVXCf24JG9/rASOzzrmAfpyAskhpdXAs+nXqSO1X3wFt5mZFeXDUGZmVpSThZmZFeVkYWZmRTlZmJlZUU4WZmZWlJOF2V6QtGOIz787vYoeSRMl3SzptbSK6GOSjpVUk26P64tmbWxxsjAbIWndocqIeD296xaSK9EXRcRikoq00yMpQvhb4NxMAjXrg5OF2SCkV8j+k6TnJa2RdG56f0VaMmKtpHslrZB0dvq080ivsJd0IHAscGVEdENSiiUi7kvbLk/bm40KHuaaDc5fAUcBRwLTgackPUZSSmUBcDhJBeB1wK3pc5aRXFEMsBh4NiK6+nn954GjyxK52SB4ZGE2OCeQVI/tioiNwKMkH+4nAL+MiO6I2EBS0qPHHGBTKS+eJpEOSZOGOW6zQXGyMBuc/hZfGmhRpjaSOleQ1CM6UtJAf4O1wM5BxGY27JwszAbnMeBcSZWSZpAs3/kk8L8kixtVSJpFsmxnj3XAQQCRrCWyCvjHvKqniySdkW5PAzZFxK6R6pDZQJwszAbnP0iqfz4HPAT8Q3rY6Vck1U2fB24mWcmsJX3OffROHheSLGr1qqQ1JOtb9Kw98HFgRXm7YFY6V501G2aSJkay8to0ktHGsojYIClHMoexbICJ7Z7X+DXw9Si+TrzZiPDZUGbD715J9UANcHU64iAi2iRdRbIm8lv9PTldOGm5E4WNJh5ZmJlZUZ6zMDOzopwszMysKCcLMzMrysnCzMyKcrIwM7OinCzMzKyo/wepsDHju8gQ/AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fd5b44b65f8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "accuracy_s1 =np.array(accuracy_s).reshape(len(C_s),len(gamma_s))\n",
    "x_axis = np.log10(C_s)\n",
    "for j, gamma in enumerate(gamma_s):\n",
    "    pyplot.plot(x_axis, np.array(accuracy_s1[:,j]), label = ' Test - log(gamma)' + str(np.log10(gamma)))\n",
    "\n",
    "pyplot.legend()\n",
    "pyplot.xlabel( 'log(C)' )                                                                                                      \n",
    "pyplot.ylabel( 'accuracy' )\n",
    "pyplot.savefig('RBF_SVM_Otto.png' )\n",
    "\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将结果打印出来，看起来更直观"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
